I don't think there's much room that isn't being exploited by this SSD expert streaming stuff.
There may well be headroom within the software/architecture of LLMs but once you get down to streaming tokens, it is kind of running into the physics?
There may well be headroom within the software/architecture of LLMs but once you get down to streaming tokens, it is kind of running into the physics?
But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.