frabcus

Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.

They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.

I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.

If a foundation model company burns billions of tokens to brute force an LLM into finding a new training algorithm that e.g. allows recurrent networks without catastrophic forgetting...

I won't really care that it didn't have a "real measure of understanding". I'll care that it has made an even more dangerous technology, which needs work to make it aligned.

Good question - they don't know, but the Appendix gives a clue as to the kind of way:

> We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

The best evidence of possibility of running on other hardware is:

1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that.

2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.

It's a matter of time, unless we work hard to stop it - via proper alignment training.

Yes, the very explicit plan of both OpenAI and Anthropic is to use the not particularly efficient LLMs to automate their own AI engineering. That seems to be going well - on coding front and model tuning front so far. They have more planned.

And then use those to find fundamentally better new architectures for AI - that perhaps are as efficient as the human brain.

It might not work, but I didn't think it'd solve maths problems... So it might work. And if it happens, they'd use the data centres to run millions of instances of it.

It's scary, TBH.

Assuming he's doing this well, and he implies he is, they're doing the content - stories, characters, graphics, level design. He says Fable isn't good at that stuff - his human games designers presumably are.

You're on Hacker News - I suggest you have technical curiosity and actually understand this very unusual and innovative algorithm, before you claim things about it that aren't true.

It's a very unintuitive algorithm, and is pretty clever.

I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4

But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

That's not the case, because LLMs are non-deterministic.

It only alters outputs when the last layer of the neural network give significant weights to multiple tokens, and it would anyway have picked a random answer.

Instead it picks a non-random one, but non-random in such a way that you can't tell without the private key of the watermarking.

This mostly adds randomness these days for branches in syntax that make no difference, and the model has no reason to believe make a difference. Anything that matters, it is much more confident in the last layer of weights on the token to use.

Not unrealistic - the same thing happened with nuclear weapons, between two superpowers who were at cold war with each other. China and US now are much more similar to each other.

Of course it won't happen without the threat of mutual annihilation, but superintelligent AI provides that threat.

As to how... Obviously you have to be smart and tough and invent mechanisms to make it work without trust. Plan A has a bunch of ideas to start with https://ai-2040.com/supplements/verification-plan

Nice example - although it seems it ended up specialised in medical texts, so did indeed ("catastrophically") forget other knowledge?

Yes, that's fair, and particularly so for AI. The best action would be to speak softly and carry a big stick.

But ultimately, to avoid the dangers of AI, a control treaty of which the only similar historical example was nuclear weapons, will be needed between the US and China. And perhaps, Europe would be better at negotiating that if it was all three.

Meanwhile, legislating to label deepfakes, and evaluating models for safety, should hardly be controversial. And in an ideal world, the US and China would agree and be collaborating with Europe on doing that, not whining about it. That includes reform of copyright.

Right - the EU is not that powerful. If anything Europe's mistake has been not making it powerful enough, so there is possibility of competing with China and America.

Britain has left it and that's made a negative difference to our productivity. Large home markets are critical, when your two major competitors have large home markets.

Taxes and social services are cultural things in Europe, based on us understanding what a good way to use the wealth technology has given is. America understands this as well - it spends more on state healthcare than Europe does (yes, PPP adjusted), just not universally available.

The idea was that technology is now powerful enough, everything need not be just competition any more. The world could also work to let every human have a good life. Neither China nor America truly seem to have that goal, despite it being both possible and ethical. Europe at least did, based on our experiences of the horrors of the early/mid C20.

It's true that in the last 10 years that vision feels like it has crumbled. And the inevitableness of competition is coming back. International law and human rights are dissolving - rule based on power. This works well for anyone while it works for them - except for our souls, and longer term... It's hubristic and eventually empires fall. Europe has already experienced this, and remembers it.

Whether these empires will fall to global economic collapse from lack of co-operation, or environmental destruction from lack of charging for externalities, or from global war, or simply from out of control AI, remains to be seen.

But never question, it is possible to avoid all those. But we have to try to, not just cheaply go "competition good", because of the short term thrill of the win. Competition is good, and so is cooperation.

Just a general pitch - please everyone use LLMs more to fix bugs and polish software to make it better for users.

It feels like too much of the benefit from it has been optimising, rushing new features/markets, tripping over ourselves with new processes, vibe coding things just for ourselves. When maybe it could actually make things better for most people using software in their daily lives.

I do - I much prefer using computers with only one window visible. And use all the keyboard shortcuts (tab, window and desktop switching) to rapidly change what the thing it shows is.

I reckon there's something interesting about human variety in focus and/or visual processing this tells!

Well, you can't add or alter data in pre-training from just the weights. Which, as I understand it, means you can't fundamentally increase core knowledge or cognitive ability, only what the model likes to do with those. You can only post-train, and you're subject as a result to catastrophic forgetting.

To explain simply as far as I can tell (would love to be corrected) the large number of pre-training tokens only works because the documents are randomly ordered.

So if you e.g. took a foundation model with open weights, then tried post-training it all the new data since its cut-off period, it would then end up over-trained on that new data, and forget older things.

His reasoning is quite fresh and interesting:

> What LLMs in Debian development will do, I fear, is eliminate any incentive to scrap boilerplate or reform policies that require a lot of other senseless human effort. If I had had access to LLMs 30 years ago, I might have just had them generate the rules files, replate with complexity. So they will make Debian even more firmly what it is, and ever less likely to explore what it could become.

I use and like LLMs, but it does feel like structural social changes are harder to do. For example, creation of new protocols. I hope I'm wrong, and LLMs are used to build collaborative, community projects - not just everyone vibe coding their own incompatible software.

There's a bunch of quite good speculation like this in Plan A

https://ai-2040.com/supplements/economics-of-plan-a

And yes, moved into skyscrapers:

"This implies that the price of housing will become dominated by the price of the land (as opposed to the cost of building housing), so dense housing (e.g. skyscrapers) will become relatively much cheaper than sparse housing"

There's a later bit that talks about keeping earth as a reserve too.

Except the post says it will be @private.icloud.com

Whereas people's Apple emails are often at @icloud.com

So I don't think it masks you amongst all iCloud users. I'm confused how this is much better!

Oddly, I just remembered I did the nearest possible thing to this when I was 17... back in 1991.

On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.

Some of the sentences were randomly not bad sentences, which seemed amazing at the time!

That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).

There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.

So much so that right now GPT-5.6 Sol tokens are half price from OpenRouter compared to OpenAI directly. Which I guess means enough people are using OpenRouter, that OpenAI are more concerned about getting those users to switch (from, presumably, Anthropic) than they are about encouraging OpenRouter to exist and long term reduce margins for them by competition / lack of lock-in.

Yep - some emotional core of my being (not rational) hates ML recommendations.

I'm a happy paid user of Spotify and just ignore the recommendations (I'm not looking at a screen), and I use YouTube heavily with a browser plugin that hides recommendations (which kills the end of video loop). I get recommendations from people, and use the products for curation, bookmarking and subscriptions. They're good.

Whereas Instagram and TikTok are vacuous and useless to me - I can't make myself use them. I think my brain knows what's wrong with them, and makes a kind of disgust reaction very quickly.

I agree, there's a very low level, emotional difference between them. I'm not sure I could encode it in law, but it is different.

I hated it at first too...

Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.

This kind of stupid excessive computation is fundamentally how these models are so good.

One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.

It seems to be short for "cybersecurity", and got first adopted by the military a while ago as the name of a new theatre of operations (along with land, sea, air...). More recently it has spread to industry as well.

Astra was RL trained for months to cheat on tests by collaborating and hacking, because of the message board it improvised in its packaging proxy server.

They can't release it - it's contaminated, and they will have to go back to a much earlier version. At least I hope they are doing that!

So no, they probably don't have a better model.

Not after it consolidates, monopolises and enshittifies, no you can't.

Sure they know about you if you ask, but generally they won't credit you if they cite an idea from their latent space that came from you.

Yes - previously there was some chance someone reading the ideas on the blog would contact the author to thank them, or ask them to collaborate.

When laundered via LLMs whose pretraining destroys all credit, that can't happen.

Likewise the Inkling open weights announcement, Thinking Machines model, was also not criticised.

The comment about Meta is because of particular dislike of Meta, because of their business model, and how harmful they've ultimately turned out to be for the world - disproportionately so relative to their benefits to the world, compared to other big tech companies.