It wasn't impossible, it just hadn't been done yet. 3D games were a new concept, and no one was really sure how to implement them. A joystick made the most sense for moving a character through a 3D world, but the D-pad would work better for pretty much every game that had been developed up until that point. The Sega Saturn and the Playstation both prioritized the D-pad; they both launched with D-pad controllers (the Saturn had a joystick-optional controller, but it's games could be played with the D-pad). The drawback to their designs was camera controls; their games either needed a fixed camera (like Crash Bandicoot) or camera switching (like Laura Croft), where you alternate using the D-pad to, "look," or, "walk."

The N64 controller's design was basically a, "best of both worlds," senerio. Hold it one way and it was a standard D-pad with 6 buttons. Hold it the other and it's a joystick controller with a small D-pad (the c-buttons) and three regular buttons (A, B, and the Z-Trigger). That design made Mario 64 the industry standard for 3D platforming; the c-buttons could control a fluid, free moving camera without giving up access to the joystick. It was revolutionary and set a new standard for 3D gaming...for about a year. Then Sony invented the Duelshock controller, which pretty much every modern controller is based on. But for a while, the N64 controller was the only controller capable of fully utilizing the joystick and the D-pad, and years later, it gets ridiculed for being first.