Isn’t the KV cache a huge component of the overall costs, and wouldn’t this make this problematic? Cache writes are an expensive part of LLM tokens, even more so for caches that have a long expiration time.
"Low-priority" inference tasks like that, though, would never need to be cached. Again, who cares if it takes 12 hours? If you didn't initiate the prompt, then what do you care? You just randomly get notified when something is already ready for you to take a look at.
I think reality is much more nuanced than what you’re sketching.
I mean, at the end of the day, either inference providers are having difficulty filling batches, in which case there's opportunity on the table to somehow fill the batches with discounted, time-insensitive, "background review" style work, or they don't have that difficulty, in which case it's a moot point. Exact cache architectures are beside the point.