It would still be irritating, not because being gay is bad, but because they think you are something you're not. It's just incorrect.

If I said "I thought you were from Mars" it would be... kind of annoying, in the average case.