"wars", or really just disputes in general I guess since culture war is metaphorical, don't really work that way. It only takes one side to start one, and then the other side must either resist them or cede what the other wants, there's no option to just opt out of them without mutual desire to do so, and if someone has started one against you that desire obviously isn't there.

The issues considered "culture war" aren't inconsequential, they affect the lives of real people in material ways. Thus, trying to dismiss them as a distraction just means accepting harm to some people. If one wants a decent society, it is necessary to win both.