I’m a man, and I feel like people judge men for wearing “weird” clothing more often. However, I think I could pull it off

Start off with a poncho. You'll feel weird for a bit but then will notice that nobody gives a shit, actually. Switch to a cloak and the same will happen.

Fun fact: I saw a dude rocking a cloak about a month ago!