I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising.
Making qwen better at math to me was a lot more surprising
Again, very impressive from a technical point. But language is so flawed to do anything creative. It just turns out all looking the same and kind of boring
> If you consider how impossible the startup journey is, it stands to reason that only through intensity can something be created out of thin air that grows to be worth billions of dollars in just a few years.
It is sad to me that this is the benchmark for so many people. It is only a success if it is worth billions.
Can someone explain this architecture a bit more in depth? They say the pointer head scores the hidden state at each option against the hidden state of the answer. But the LLM produces hidden states per token, so an option can span multiple tokens, no?
The tech has always been mature enough but simply too expensive. But since combustion engines are sold out for the next years, data centers are rushing to get anything they can. So the extra millions for power generation will be a rounding error
This feels very useful for training agents but almost completely useless to humans? Do I really want to go back to every change I made? Whatuse would that be?
To me it feels so weird that people are trying to push these model for online shopping like "here is how this dress/shirt/pants would look on you". But these models will always make the clothes fit your body and show you in flattering light and so on. How the actual garment fits is still as elusive as before these tools
The "climate" cares little about carbon intensity of labor or dollars. It cares about absolute tons of green house gases in the atmostphere and if you use more to produce more you still use more.
What prevents a bad actor from posting "easy" problems to the board, solving them and getting quick reputation. Then as a validator they can easily validate their own malicious changes to someones software?
I always wondered how these models would reason correctly. I suppose they are diffusing fixed blocks of text for every step and after the first block comes the next and so on (that is how it looks in the chat interface anyways). But what happens if at the end of the first block it would need information about reasoning at the beginning of the first block? Autoregressive Models can use these tokens to refine the reasoning but I guess that Diffusion Models can only adjust their path after every block? Is there a way maybe to have dynamic block length?
Does anyone know what kind of RL environments they are talking about? They mention they used 15k environments. I can think of a couple hundred maybe that make sense to me, but what is filling that large number?
With the rough numbers from the blog post at ~1k tokens a second in Cerebras it should put it right at the same size as GLM 4.7, which also is available at 1k tokens a second. And they say that it is a smaller model than the normal Codex model
I think this says a lot about the business approach of Anthopic compared to OpenAI. Just the vast amount of free messages you get from OpenAI is crazy that turning a profit with that seems impossible. Anthropic is growing more slowly but it seems like they are not running a crazy deficit. They do not need to put ads or porn in their chatbot
I feel like this comes from the rigorous Reinforcement Learning these models go through now. The token distribution is becoming so narrow, so the models give better answers more often that is stuffles their creativity and ability to break out of the harness. To me, every creative prompt I give them turns into kind of the same mush as output. It is rarely interesting
They trained it in 33 days for ~20m (that includes apparently not only the infrastructure but also the salaries over a 6 month period). And the model is coming close to QWEN and Deepseek. Pretty impressive
You cannot negotiate with a bully. The EU should have never backed down so easily before. I hope someone will soon find some balls and not let the US walk all over everyone
Is it actually possible that nvidia chips will have 50TB/s bandwidth by 2028? Right now it shows they are at 8 TB/s. To me it seems the Nvidia forecast is a very, very optimistic exponential. Nontheless, Huawei not matching the scale of production seems to be the biggest hurdle
To me, this only makes sense on a word level not sentence level. I can understand that words, especially those that are older, have evolved because of energy and comfort constraints of our physiology. But to extend this to sentence level is a rather big step. I would suppose it works for simple, short sentences that had to be efficient in the past. But imagine sentences about computer science where most words are rather new and have been chosen by arbitrary rules.
To me it would be interesting to see, if this hypothesis holds when applied to longer and more complex sentences and "modern" words.
That is partly true. High population density means a lot of roof area. Solar is perfect to put on roofs, you need no extra land. It is basically free (save the investment of the panels which pay off quickly nowadays)
If we think of every generation as a compression step of some form of information into our DNA and early humans existed for ~1.000.000 years and a generation is happening ~20years on average, then we have only ~50.000 compression steps to today. Of course, we have genes from both parents so they is some overlap from others, but especially in the early days the pool of other humans was small. So that still does not look like it is on the order of magnitude anywhere close to modern machine learning. Sure, early humans had already a lot of information in their DNA but still
".. Claude estimates that AI reduces task completion time by 80%. We use Claude to evaluate anonymized Claude.ai transcripts to estimate the productivity impact of AI."
What is this? So they take Claude and ask how much do you think you saved on time here? How can you take this seriously. ChatBots are easy to exaggerate, especially about something positive like this.
We are so much more productive and efficient, so we should even work on weekends to be even more productive.. and then they make an ID System that has been solved for ages with simple passports