But that's my point - throw the "background code review" stuff out of cache, or into object storage, or whatever, just something to draw upon to fill inference batches opportunistically.

I mean, at the end of the day, either inference providers are having difficulty filling batches, in which case there's opportunity on the table to somehow fill the batches with discounted, time-insensitive, "background review" style work, or they don't have that difficulty, in which case it's a moot point. Exact cache architectures are beside the point.