They tried to aim for a more premium console with the N64 and the Gamecube, and the result was Sony ate their lunch.

To be fair, they shot themselves in the foot by not using CD-ROM and regular DVDs respectively. The Nintendo 64 was a very anticipated piece of hardware, but they took too long to release it and then many companies jumped ship due to said lack of CD-ROM, how long they took and cheaper licensing too most likely.

They never actually competed on equal terms thanks to Yamauchi's stubbornness.