For years I thought "bugger" just means "to annoy" or something like that, so I would say things like "This buggers me" or "You don't have to get so buggered about it."

Boy am I embarrassed...

I've never encountered that word used with a homophobic intent, as far as I could tell. If you wanted to express that thought in my youth, other words beginning with B would be far more common. Naturally I found them as odious then as I would now.