The top models also seem to have inconsistent performance depending on the time of day and how far we are from the next release.
The top models also seem to have inconsistent performance depending on the time of day and how far we are from the next release.
Even with minor automation I feel like I can watch OpenAI and Anthropic engineers fiddling in real-time. Tuesdays behaviour changes by Thursday, 10AMs production isn’t possible at 11:30AM. Nutty.
Meatspace fluctuations are predictable, quantifiable, and manageable through planning. The changes to hosted model behaviour are not.
Literally, the prompt terms used to manage document reading can change multiple times within a day. MS and Oracle provide decades-long consistent interfaces in their data products.
Which is what I suspect the providers are doing to fit more inference on the same amount of hardware over time.
https://marginlab.ai/trackers/claude-code-historical-perform...
There were at least a couple of these degradation trackers.