Just whether those parts of the network see activations during trivial coding tasks (I’d have to assume they actually do sometimes, depending on the input prompt)
1,200 karma · joined November 4, 2022
Just whether those parts of the network see activations during trivial coding tasks (I’d have to assume they actually do sometimes, depending on the input prompt)
The really expensive part of inference - the compute and memory traffic required to perform prompt processing and decode - is paid for roughly once per token in a multi-turn session when prompt caching is working properly.
The difference to the provider's costs between:
- Submitting 100 back-to-back requests, each adding 1k tokens to the context
- Submitting 1 100k token context request
Is negligible. Yet the consumer is going to pay a lot more in the former case for all those repeatedly submitted input tokens that you get charged the cached rate for (but which required no additional compute on the provider side).
DeepSeek V4.1 has an insanely optimized KV-cache that takes around 900MB per million tokens, making it feasible to move to disk between requests. Makes sense they're cheap
I’ve already seen that trick snuck in by newer models. “I won’t use em-dashes; I’ll use semi-colons!”
I’ll absolutely be giving this a go once weights land (a QAT FP4 experts release would be most appreciated)
This is still a welcome model, even if I won't be running it at home
It's unsurprising and perhaps even desirable that different LLMs return similar results to an identical prompt.
If you want variation, prompt for it.
Using the output of a “decision” model without insight into the reasoning for a given decision seems very trusting.
If the output of the model is its reasoning trace, and you simply feed that back into the model again at inference time instead of outputting it - then it is by definition hidden (but I would expect you could pull both this trace and a further-down final output trace out)
V4 flash and V4 pro feel very similar, which would make sense if they were pre-trained on largely the same corpus.
All that would suggest to me is that V4 Flash is capable of absorbing the data they’re throwing at it, and we’re still nowhere near the data limits of their larger 1.6T model
Focus on the models
I still enjoy seeing the pelicans.
Edit: Ok, max effort made a darn good pelican.