The N64's RSP was quite exciting for the era because of the high degree of programmability. As it turned out, though, almost nobody did anything to program it. They used the microcode that shipped with the devkit. And since the N64 was the odd one out of the pack in other respects, it never really took off in terms of developer support.

So Nintendo's biggest lesson from the N64 was, in fact, that they needed to make boring hardware, because weird stuff wasn't going to be adopted anymore.

Sony learned the same thing almost a decade later with the PS3.