It's a hard problem that token space only partially solve, because with filling the token space to the brim with documents will dilute the instructions in the prompt.

One may argue that we'd need a separate channel for the instructions, but it would make harder to do anything but zero shot question and answer.

I think it's likely that systems with metadata for tagging sources, tasks separately from the human and ai interaction will eventually emerge. The gptchat api already works somewhat that way, but it isn't following the structure that we'll, probably because it was a later tune and not something built in the model from the beginning.