I think you mean that the problem is with extremist, not with religious people. Also, people where I live do oppose them because being an asshole is generally against Islam. (I live in a Muslim majority region)