One weird thing you missed is the "length of credit" though. Let's say you get your first credit card somewhere as a kid, and now you're 40 and haven't had anything else. Suddenly, that bank wants you to pay a monthly fee to keep your account open. If you get a new credit card somewhere else, and cancel your kid-card, your score is going to get hit quite a bit, no?
If you did it "correct", you'd have 10+ different cards lying in a drawer at home to tons of different credit agencies.
It's all an insane game that you have to play, despite the fact that it can usually be played "for free".