Hacker News Reader

So with 128K context window, if you actually input 100K it would cost you:

Input: $0.01 per 1K tokens * 100 = $1.00

$1.00 per query?

Given that each query uses the entire context window, the session would start at $1 for the first query and go up from there? Or do I have it wrong?

View on HN

If it truly is GPT-4+ with a 128K context window it's still absolutely worth the high price. That is literally 300 pages. However if they are cheating like everyone else who has promised gigantic context windows then we are better off with RAG and a vector database.

It would be $1 for each individual API call, if you were continuing the conversation based on the same 100K input. ChatGPT is stateless.

Right, so that adds up very fast.

This is a sad fact, and one which they should have implemented a fix for.

We know medium term memory works. Sentence transformers and everyone playing with pooled embeddings knows what it is because they're using it. I should be able to map my previous history to a smaller number of tokens using embedding pooling to give a notion of a lossy "medium term" memory independent of RAG.

Reply on news.ycombinator.com