I imagine OpenAI is not far behind in expanding the context window of its models. The LLM companies have access to the same techniques and - in my estimation - are just choosing to focus on one aspect or another to address different market needs. For instance, Claude 2 clearly focuses on maximum context window size at the cost of speedy inference and, presumably, inference cost. By contrast, OpenAI seems to be focused on speed and low cost (GPT-3.5) and accuracy (GPT-4) rather than maximum token length.
For example, if I put instructions first, then a lot of context, it would forget the task, so instead put a small explanation, then context, then question, but still it would seem to sometimes loose the plot.
For feedback on prose though, I found it Claude very strong. It's nice to be able to give it half of a book, in text format and have it guide you to important parts and summarize sections. Some things, like what are the themes of this work, what's surprising finding, don't work well with vector DBs.
[1] https://www.pinecone.io/blog/why-use-retrieval-instead-of-la...
The tasks I'm doing might be uncommon. Here is a build script in some language you (the llm) understand, and I want to get it translated to a language you don't. And so the context is example conversions and documentation.
I've also had some luck with things that boil down to "Here is a very large style guide. now how would you improve this code?". Or "here is a number of examples of feedback on writing to conform to a style. now generate the same on this new input."
I found the large context windows and Claude to work quite well in those examples. But, if its possible, breaking it down into multiple steps with less context somehow and using GPT4 even better (though more work).