I'm 39. These days I explicitly work out to be in good shape for sex. I want to look good when I pop the top, and attracting partners is a core goal of mine.

I've never in my life been hurting for confidence, and was confident when I was 40 lbs heavier - I just looked worse with my shirt off.

If you're chasing low-commitment sex, at some point your physical attractiveness does indeed matter.