So, I'm a moron.

I did it my whole life until I took a stats course and it was like the first this the prof went over and I saw it was pretty obvious once it was pointed out to me but took someone having to point it out to me for me to see that mental blind spot