To add to that, I think this may have been the game that started that trend. There was always a difficulty ramp, but the sudden spike level came later in the series. Any Mario platformer these days has a crazy difficulty spike near the end.
I mean, New Super Mario Bros. 2 had an entire DLC called "impossible levels." And don't get me started on The Final-Final Challenge in Super Mario Bros. Wonder.