I used to think that until I got a freelance gig for a custom dress shirt company. The cut of the suit and the shirt make a difference.
For example, a button-down collar is more working class since they’d be in situations where the collar could pop up. This makes it more casual than a collar with stays.