I think that people who hate math mostly just hate the way it's taught and have mentally blocked themselves from ever trying again. I also think it's cultural. There's no reason so many other countries have absolutely no problem with teaching what is considered very advanced math concepts to young students while many in the US struggle to even pick up basic concepts in secondary school and college. It's learned helplessness.