Had that happen two years ago. Buncha yellow jackets made a nest on the outside of a metal egress and got me twice while mowing. After trying foams and sprays, I found boiling water to work quite well. Although maybe they were ground bees after all, esp with your mention of agitation when mowing. I even reached out prior to the extermination attempts to local apiary specialists to confirm what they were.

if they are what im thinking of in the fall they become sorta dicks.