For understanding:
What would be a suitable input length in your oppinion?
And why isnt this a good one: Are real-life queries shorter? Or longer?
If i count one word as a token, then in my case most of the queries are less than 128 words.
What would be a suitable input length in your oppinion?
And why isnt this a good one: Are real-life queries shorter? Or longer?
If i count one word as a token, then in my case most of the queries are less than 128 words.
It's not just the query (if you're running a chatbot, which many of us are not). It's the entire context window. It's not uncommon to have a system prompt that is > 512 tokens alone.
I would like to see benchmarks for 512, 1024, 4096 and 8192 token inputs.
If I understood that correctly, context length is something like session storage or short term memory. If it's too small the AI starts to forget what it's talking about.
I tried to look for some service provider to publish this kind of metrics, but haven't found any.