ismael_rr

they started their massive ai capex over a year ago with aquihiring scale ai, so I think it makes sense that their investment is paying off now. in total they've spent in the high 200 billion on ai, though I think most is likely on datacenters.

I suspect that 6 sol was a better version of 5.6 terra (and note that in the 6 sol and luna release, they took out terra, and price 6 sol at 5.6 terra pricing), then the backlash from lesser capabilities made them roll out 6.1 sol as the actual 5.6 sol - size modee.

Same with the US legal system protecting Anthropic from copyright laws from all the books and other stuff used for pretraining...

Distillation is also a broad term - I think most specifically, it refers to training a smaller model on a larger/better model's full output token distribution rather than normal pretraining, which only can access the next token in the data that was actually used.

It's also used to describe the SFT bootstrapping for posttraining, which is what people generally refer to as Chinese labs "distilling".

I would almost guarantee that smaller US frontier models (ex Luna/Sonnet) are distilled from their respective large models.

haha was also going to mention this at the risk of pedantry

Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa

In my experience, it feels like this problem has been mostly solved. When I interact with a chatbot, I see it often uses web search to answer my questions and hallucinates much less then they used to. Furthermore, I suspect a lot of gains are merely from the system prompt instructing the chatbot to use web search to verify all facts it says.

I agree that most AI use in terms of users may not be for coding or agentic use (everyday people are asking chatgpt for something or looking at google ai summary), but with respect to AI usage, I speculate that the vast majority of usage is coding and agentic because they're super token hungry.

In terms of the value proposition of AI replacing knowledge workers, all value is in coding agents (coding agents as general agents).

for you tennis players out there, I'm working on RallyClip, an app for free tennis match segmentation. It's a free and open source alternative to SwingVision:

github: https://github.com/iroblesrazzaq/RallyClip website: rallyclip.app

I'm working on model performance and an iOS app right now; would love feedback on the macOS desktop app.

It was fun! I like how unlike the nyt games one, you can get additional information from the edges' existence. Would advise removing having both plural/singular forms of a word. Great work!