I think the problem is not necessarily being a tourist, but egregiously bad tourist behavior.
I have family in, and we're moving to, a tourism hot spot. I honestly don't mind about 95% of the tourists. I even like a lot of them, since the fun ones are great to have around and you can learn about people's lives in so many different places.
The relatively small fraction that acts like total shitheads is what really stands out to me.