You could do multi level summaries etc but yeah this is all just band aids around token limits.
The problem is that humans have continuous information retrieval and storage where the current crop of embedding systems are static and mostly one shot.
This weird leaky memory has advantages and disadvantages. Forgetting is useful, it removes garbage.
Machine models could vary the balance of temporal types, drop out Etc. We may get some weird behavior.
I would guess we will see many innovations in how memory is stored in systems like these.