I just checked my VP40 I have slight right left play also. I think this is normal. I would look at changing the grip panels and back strap.My 40 came with medium all round,I assume yours did also. I just changed the rear to small and the left grip panel to small. I have not shot it yet to see how that works. I have 350 rounds down so far. As far as my hand size I can wear L gloves but hey are very tight XL fit more comfortable. Depending on hand size and comfort you may want to experiment to see what works best for you. You say your groups were large, were they center of target?
I expect you are right, some were dead on, many were a few inches left and low (I'm right handed and often have those issues with a fairly new gun till I get the grip and the trigger down) I did change grip panels from last time at the range to large everywhere except small and the left hand side, might try all large.
The odd thing was in slow fire one would be spot on and the next would be low and to the left, no matter how many grip changes and and trigger finger changes and slow fire this would repeat, one on, next low left about a few inches left and down. Change grip and finger position, one on, one left low. This was all slow fire, normally in slow fire once I find the grip and trigger finger position I am spot on till I law it down and have to find the correct grip and trigger placement again till I get used to the gun. (All about 15 yards)
On thing I found last night is the "wall" seems stout and long for me, I could see the front sight wiggle. And as at the range, one might have not wiggle and the next would without changing grip. Since the wall seemed stout and long and I only have perhaps 300 round through it I dry fired it 800 times last night and that seemed to smooth it out quite a bit, I am trying large panels all the way around next week and taking all the panel to the range.
Might try gloves, any other advice is welcome, just never had a firearm in slow fire mode that did not hold a solid pattern through a mag with no finger or grip change. Fast fire I often go slightly low and left but that normal for me when I push it and still well in the zone I want for self defense.
Thanks.