While I don't doubt your stance comes from a history of trauma, policing any kind of identity in this way causes real trauma to others. It causes a pervasive sense of isolation that is antithetical to feeling supported and secure and puts a check on a person's ability to participate in their culture. Your lack of comfort does not mend leveling the playing field of stripping away the comfort of others if it is being expressed peacefully.
Bans also very become a very fuzzy line. Most holidays are based off of religious festivals that are widely participated in by the secular and non-secular alike. Once someone starts making exceptions because a wide number of people like a specific one you start creating an artificial canon where minority cultures are oppressed while a narrative of "dominant culture" is allowed giving certain religious traditions cultural supremacy. For example people inside the Church have been trying to get rid of the multitude of pagan festivals that were rebranded as Christmas for eons. They ended up just rubber stamping it because taking away something beloved doesn't go well. In a modern context you could try and rebrand Christmas to a non-religious holiday... But good luck. It's layers of Christian over Pagan imagery and traditions fused into a gastalt religious melange. Any governing body that has tried to get rid of it before has spectacularly failed and leaving it be would quickly become a symbol to people who come from places with different dominant partially seclarized religious traditions that they remain cultural outsiders who don't have the nessisary concensus to participate in public. It would translate directly into supremacy narratives.
It's healthier for a society by far not to police the range of peaceful human expression and connection. People deserve to see themselves represented and connect with each other without needing to act like undercover spies in hostile territory.