Hacker News Reader

Comment by iandanforth | Hacker News Reader

FWIW The default context window of GPT-4 via ChatGPT is about to change to 32k.

Given the number of times it just fails with large prompts on 32k contexts, I'm not sure if they'ready for this. In my experience, if you're consuming 20k+ tokens failure rate is more than 50%.

Well over 50%, at least via the api, for me.

This would be great if true. Any source for this?

https://twitter.com/DataChaz/status/1719660354743976342

that would put them significantly ahead again, for my use cases

We will eventually increase the Phind Model to 100K tokens -- the RoPE embeddings in Code Llama were designed for this.

> the RoPE embeddings in Code Llama were designed for this.

The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE (e.g. position interpolation) are post-hoc tweaks (with optional tuning) to an entirely vanilla RoPE implementation.

100k tokens and good ide support would be great. Copy pasting back and forth with browser and IDE is kinda annoying and you always miss some context. I think model is now good enough but what is kinda missing is good developer experience eg what to load in that context window and how model integrates to IDE. But this is kinda missing with copilot and chatgpt4 as well.

Is it “100k” or really 100k there are so many ways to do context, I remember seeing 100k before but it was doing some cheap trick to get it

What about ALiBi and Sliding Window Attention?

Additionally Apple researchers seem to be playing with "Attention Free" variants.

Source?

Reply on news.ycombinator.com