Of course this will only work if your user's state can be captured within the 4096 tokens limit or whatever limit your llm imposes. More if you can accept forgetting least recent data. Might actually be OK for quite a few apps.
I suppose you could divide and conquer with smaller parts of the algorithm, but then we'd need a "meta AI" that can keep track of all those parts and integrate them into a whole. I'm sure it's possible, don't know if it's available as a solution yet.
Both less and more than GPT, because humans can learn from limited input and also we have a lot of tricks for escaping our horribly limited context size. GPT probably has a larger context than humans, but it’s worse at everything else—to the degree that’s comparable.
I wouldn’t bet on that changing soon. I also wouldn’t be on it staying the same.
I tried similar prompts on various data structures. If you reissue the request sometimes that completes.