Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.
What would be a suitable input length in your oppinion?
And why isnt this a good one: Are real-life queries shorter? Or longer?
If i count one word as a token, then in my case most of the queries are less than 128 words.
If I understood that correctly, context length is something like session storage or short term memory. If it's too small the AI starts to forget what it's talking about.
It's not just the query (if you're running a chatbot, which many of us are not). It's the entire context window. It's not uncommon to have a system prompt that is > 512 tokens alone.
I would like to see benchmarks for 512, 1024, 4096 and 8192 token inputs.
I tried to look for some service provider to publish this kind of metrics, but haven't found any.