Perhaps they are preparing for a switch to usage-based pricing for everything at some point and are thinking of ways to burn tokens? Not to be conspiratorial but many companies think like that, and tokens = money at this point.
For example if you ask Claude chat to write something of a certain length, it won’t, it’ll just reply at the same length as everything else. Why? Because they’re billing at a fixed price. They don’t want you to get a whole essay as an answer because you’re on a plan.
Claude code on the other hand, which is supposedly the same model, will happily write out a billion lines because it is billed by the token.
If you read the model specs, they have a maximum output length. If you are running into that it might simply be that you are seeing the difference with a harness that can allow it to take another turn automatically and build up the output in a file vs the chat which may not be able to, though they do get some tools there.
The response output length limit includes the thinking tokens.
It's going to be a mess very soon. A huge mess. A huge incremental cost mess.
You're being hooked on the good heroin and they're going to charge you for the shit stuff later.
A man can dream, right?
Only partially joking there.
said that as a joke then remembered how all the drone weapons being used in ukrane/elsewhere are in a way just computer chips with explosives and motors... so ya you may have a real point.
It'll go from a monthly usage plan to a per-token usage plan with minimum commitment. Same as AWS pricing did.
Customers pay for things that are useful. If the token drifts from being a reasonable proxy for what is useful, customers won’t accept being charged on that basis (or jump to the best trade).
If I have an employee who works 1 hour and builds me a beautiful cabinet and another who builds one in the same time, with a door falling off, the one with less productivity per hour (token) is not getting a call back.
Depends on the definition of useful. In a corporation "useful" is "checks regulatory boxes" - it doesn't need to actually be useful, it must just pretend to be useful. Hence all the shit customer service processes. They are useful to the corporation as in that they tick the "we have customer service" box. Whether the consultant just hangs up on you instantly after picking up doesn't matter.