Did you forget that on the N64, load times were near instantaneous?
So there literally were no "loading" times for these assets. This might not even be realistically possible with NAND flash based SSDs, e.g. because of considerations like latency.
Though directly accessing ROM memory would also prevent things like texture block compression I believe.