It's like saying AIDS doesn't kill people because it merely weakens their immune system.
Which is accurate. It's the infection or cancer that kills you, like the priest or the astrologer oppresses you. The belief might make you more susceptible to oppression, but it's not what's oppressing you.