Religion is a whole other thing. Sure, it can direct your moral compass, but you don't need it to have a moral compass. It can make a bad person act good, but it can also make a good person act bad.
I agree that in the end, it is all about the effect you actions, or inactions, have on others.