As I suspected the patch is currently far from ideal. When it was created the small stones probably looked quite attractive on the surface, but over the years have worked themselves into the soil. Luckily they're hard to spot anyway in amongst the scattering of dead leaves. Also there are other things growing in there... mainly ivy from a nearby plant, but also some holly and bits of other plants I don't recognise... which don't
look like weeds, but whaddo I know.

Raking, destoning, digging, root removing and more digging was pretty tiring, and that's just my first bit. And my back aches despite bending the knees. Either I'm going to have no back by the time I get to eat the first veg, or it'll be as strong as iron. Can you get "Gardener's back" like "tennis elbow"?
The first tree eventually came out with a huge root ball. However I was a little worried that the roots left in the patch may continue to suck up water and nutrients that are meant for the veg, so I dug as deep as I could and got as much of each remaining root as I was able. I know I'll not be able to get rid of them completely, but hopefully I've done enough.

I'm going to put some an ant poision down - it's in trap form so that they take it back to their nest - I didn't want to spray powder all over where I'm about to grow veg. And I also got some slug powder, which I'll use now, but which I can continue to use once the veg are planted.
Tiring, but a good start.
My current aim is to get all the bushes out before going on holiday in just over a week, and then covering the ground in black plastic while I'm away to help disuade any plants/weeds still there. I don't know how realistic that'll be, but it's a plan.