> Ah yes, I forgot about part X

Would something like LMCache help here?