You're right. These days, I dress too sloppily for one, but back when I didn't, I was self-conscious about it. I probably wouldn't be now, but that's also the reason I don't really care that I wear T-shirts all the time.
I still hate it that it has that reputation.