I'm a man, and I feel like people judge men for wearing "weird" clothing more often. However, I think I could pull it off. I'm a fairly attractive guy with long hair, and I got complimented for my ponytail the other day, and men just don't get compliments usually, so going with something more unique is nice.
I'll look into it.
Edit: looking at stuff, a ruana might be perfect. It's more versitile than just a cloak that can only really be worn one way.