Your 'first floor you see' way of thinking only holds up for the ground floor.

Yes, but that thinking does hold up for that case. If it really bothers you that people take different paths through buildings, then the answer is to strictly stop naming floors sequentially (third floor, fourth floor, …) and only index them (floor 3, floor 4, …). In that case, floor 0 can be wherever you want.

Which floor you encounter 2nd in a building will depend on if you go up or down, sure, but for most buildings, every single person will have the same first floor they enter. It’s ridiculous to have something called the “first floor” that is literally no one’s first floor when there’s a floor that everyone has to enter first.

If you've got a building that has only basement levels aside from the ground floor, I feel like the first basement level being the "second floor" is completely valid. The reason we usually use a different system for basement floors is because there's already a second floor that isn't the first basement level.

Meanwhile just pure rational (mathematical even) +/- X from ground floor is logic all the way around

If we want to be purely mathematically rational then we're either using the US system where the ground floor is the first floor or we're using the Russian system where the ground floor is floor 0. Either way, we need to acknowledge that the ground floor is the start of the count. You can't just say it's not a floor because it happens to be the place where you entered.