ha... I'm vegan, me sitting at a table to eat is enough for people to feel the need to defend themselves and their choices without me saying anything other than kindly answering their questions from a conversation they start. It forces one to train the skill of not engaging with people like that.
Also he's the one that started it with the (likely bad-taste) joke, not you, but good on you for taking one just to end it. Some people just can't deal with their cognitive dissonance because it would mean facing their actions not matching with their idealized version of themselves, you can't have reasonable conversations with them.