crazylogger

Selling tokens might not be the best business ever, but since starting random startups is an even worse business, you'd stick with selling tokens!

The backdrop being deepseek offering 1% (I remember it was ~1% when 4-pro first came out early this year - 4-pro is now removed) / 2% (current for 4.1-flash).

Let's say Claude Code's system prompt is updated to recommend responding in Simplified Technical English. It'll work for the current generation of models. But Anthropic will train the next generation via RL on Claude Code traces as they do today. As long as they don't change their reward design, the next generation is going to be pulled towards Claudish again, because speaking Claudish gives higher rewards, so in the end the prompt doesn't really matter.

Ideally these things shouldn't work like this but this seems to be the sorry state of RL right now.

Assuming there is an index, agent still needs to write queries against the index to get any information from it - might as well grep!

Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.

It's not about discrediting AI. We know LLM is a commodity technology like electricity at this point. If somebody in 1900 claimed they had a setup at home where they feed in electricity and cool air comes out the other end (meaning they invented AC), obviously people would want to know what the setup is, so everybody can have AC.

They should just be symlinks to CONTRIBUTING.md and other docs humans are meant to read.

The whole idea behind LLM is they use the same language and tools as us, so we literally don't design anything specifically for them.

All the Chinese reporting I see point to the second (majority) interpretation. Liang being furious about his private investor talk leaked online is the news here.

e.g. https://x.com/_FORAB/status/2081034500101017616?s=20

The goal is to test for AGI where G stands for general, that means ability to act in any environments, ideally solving novel tasks using novel tools we’ve never seen before in the world. If a specific prompt or tool design lifts a model’s score it’s a sign the model is overfitting to a particular modus operandi, therefore not general.

I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.

Certainly sounds like a "P vs NP" style conjecture that shouldn't be possible in practice, save for certain generalizations, such as "this is a cybersecurity task, we know Fable will refuse (and score zero), so we just route to K3".

> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).

Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.

The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.

Manufacturing is primarily in China - that's true for Go Pro & everyone else and almost needless to say. The point is China usually eats the design layer too, making Apple a little unique in that they survived Chinese competition completely unscathed.

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.

To be fair while almost no company publishes what a "credit" or what a 5h window in their subscription plan really is in terms of input/cached input/output tokens, Xiaomi does: https://mimo.mi.com/docs/en-US/tokenplan/Token%20Plan/subscr...

Students will just one-shot whatever lab problems professor comes up with using Claude Code.

Actually they don't even need to compete against frontier closed models, they just need to work.

99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.

In a typical agent loop your N-th LLM request naturally becomes prefix for the (N+1)-th request. As the thread grows longer, cache hit rate converges to 100% and unit pricing for cached tokens is 10-100x cheaper.

Chinese labs access Claude via API. Isn't it the black box method by definition?

Xiaomi Mimo v2.5 is my favorite alternative. Matches DS v4 Flash (official) pricing exactly and supports image/audio/video input.

Cache hit rate dominates your total cost calculation for long agent session, and it largely depends on the provider. Deepseek's native deployment is probably much better than third party in this regard. For v4 pro it's a whopping >100x price difference between normal input vs. cached input tokens.

The promise of intelligence might be larger still. By scaling and using superintelligent LLMs to write code for itself, it's possible that the whole field of robotics is just another problem you can point LLM agents at and expect to be solved by afternoon, just like one of those math puzzles. "Traditional" robotics R&D (or any R&D really) would be worthless due to abundance.

This is equivalent to buying 3 extra days of free time with 60% of your income. You want more (holidays, in this case) and you buy them with work.

They basically said "Deepseek ran 150,000 requests and here's the gist of one of their prompts". Anthropic doesn't know which accounts are Deepseek proxies beforehand, so definitely sounds like retrospective analysis of broad user logs to me.

Of course Anthropic realizes saying this straight is problematic so they said they examined request metadata, but no, I don't think they can get this kind of insight from metadata (token counts, request time, etc.)

Didn’t they all but admit they’ve been storing and actively looking at requests with this post: https://www.anthropic.com/news/detecting-and-preventing-dist... ?

If they weren’t storing, they’d be oblivious to what customers are doing, making this kind of detection impossible. What data did they train their classifier on, if not real user (distiller) traffic?

> It might seem weird for us to teach beginners Python, knowing that they’ll then have agents output other, faster languages. I see an analogy here with Chinese: Many people don’t realize this, but children in China first learn Latin characters, which they use to spell out Chinese phonetically, using a system called “pinyin.” They then use their knowledge of Latin characters to learn Chinese characters, whose pronunciation isn’t obvious from the characters themselves.

The article did explain, albeit near the end.

OpenRouter may see you fire hundreds of requests at them, but they have no idea that "these 50 requests here at 4PM are for task A", "those 100 requests there does task B", etc. So it's a shallow analysis at the "overall request shape" level.

For a single database using UUIDs, yes, it's astronomically rare. But it's quite a different thing to say that no computer system on Earth has ever experienced a UUID collision. The number of systems out there is also astronomical.

People had this "why you probably can't run a GPT-4 (or even GPT-3.5) class model on your MBP anytime soon" conversation before.

Today's LLMs are able pack much more capabilities into fewer parameters compared to 2023. We might still be at the very rudimentary phase of this technology there are low-hanging efficiency gains to be had left and right. These models consume many orders of magnitude more energy than a human brain, this all seems like room for improvement.

The right question: is there a law in information theory that fundamentally prevents a 70B model of any architecture from being as smart as Opus 4.7?

"Hi ChatGPT, propose and prove something radically new in the genre of Gödel's theorem."

How is this not just another proposed problem (albeit with a search space much larger than an Erdos problem's)?