I have a small shoulder bag but feel the same. In UK weather having a thin rolled up coat is indispensible, a water bottle, wet wipes, headphones, mobile battery, lozenges, basic medication, bottle opener etc. Comes in handy. Men have pockets sure but there's only so much you can stuff in there.

The only problem is when it gets cold enough that I might need a pullover but might not. A bigger bag might be useful.