Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens.
I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it.
Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now