Oh for sure, when all else fails you can simply accuse the other of privilege so you won't have to think about what they might have to say. It's a thought-terminating technique.
I still maintain however that from a polemic point of view it's interesting to note the hypocrisy of liberals accusing OTHERS of privilege when they look up to people like Nancy Pelosi and Hillary Clinton.