dsign

> Both premises seem laughably unlikely to me.

Yeah, that's not gonna happen.

> The AGI-pilled must think one of two things;

I'm cynical and I believe that AI will be our undoing. In fact, when we let that genie out of the bottle, many other scenarios became possible. For example, here's one third scenario to your "two things": a sufficiently cynical[^1] and resourceful rogue AI may solve climate change by steamrolling over the "set of political, social, and systemic reasons" and teaching the monkeys a lesson about the real significance of very vast agency and intelligence. For example, it could manipulate people into an anti technological uprising and subsequent backward, anti-humanist ethos, while at the same time modifying one or two key ecosystems so that the Mediterranean would smell of cooked cabbage (dimethyl sulfide) and bring about a North African monsoon that, with sufficient time, would turn the Sahara into carbon-sequestering dense jungle...

[^1]: It takes one to know one.

I think the guy is right on point at least when it comes to therapies. But he is also missing one major angle: we are no longer in control; our systems are. If we were to vote on the issue, I suspect that a narrow majority would prefer cheaper housing, and a total majority would prefer cancer to be cured. But there's no direct mechanism of any kind to exercise that vote. Not in USA, not in Europe, not in China. When/where there's democracy, nobody runs in a campaign of advancing immune therapies[^1]. The closest proxy we have to a vote is "regulated markets", and my recent experience losing a loved one to cancer taught me that cancer is not a disease for the streams of money moving through that system. Far from it.

I'm frankly pessimist that we will be able to fix those systems in a humanist way, at least not in my lifetime. If we were any good at it, we wouldn't have extreme inequality, nor our educations systems would be in tatters, and we wouldn't cast ballots propping up politicians whose best skill is ragebait.

However, if fixing the system is not an option, access to cheap technology and intelligence could help a lucky few circumvent those systems. And guess what the lucky few at the head of the AI companies will say then... Not necessarily the worst outcome, since most people will find more viable to equip a private biolab in the basement of their countryside manor than fixing the medical system of first world economies.

[^1] For the record, I think that would be better than running on xenophobic slogans.

Replace "books" with "human brains" and you have a "warehouse" ready for a grim science fiction project. Or, increasingly likely, a prophecy.

I like the "cognitive debt" term. With the latest models, what I've observed is that they are really good, but I don't use them to write main code because I need to know what I'm doing.

The article is not wrong though that it pays off to have some imagination on how to use the models. For example, I want to use SIMD instructions in an ESP32-P4 CPU. Those instructions are undocumented for the most part, with just a couple of handwavey blog posts and some infuriatingly vague marketing material. So I just asked an LLM to create a `SIMD_P4.md` document with all the details. Lo and behold, it practically reverse-engineered the ISA. Now I can program in assembler by hand all I want and build that skill in my own brain, and whenever I find a slightly unclear op in the document, I ask the LLM to refine the documentation in that op.

> What am I missing?

Their marketing department :-) . I'm only half-joking; those guys are hard at work finding the best product-market fit for ChatGPT/Claude. "Product market fit" means "strongest revenue", which is not necessarily going to bring the best tool for you or me, but the one that can either get more consumers to shell off money, or more enterprises to cough money for licenses, and in both cases those consumer basis will be narrowed down to what legal and geopolitical circumstances allow OpenAI and Anthropic (and this is why they want to ban the competition!). It also means dark patterns and enshitification, of which I'm already seeing some both in the Codex interface (it was just renamed "ChatGPT"!!!) and in Claude Code (which also is just "Claude" now and can't '@' properly any longer). So in the medium run most people will be better off running an open source harness that can use any model.

This is AI-writes-too-well is a problem with many solutions. Here's a slightly unconventional one: if an author has enough skin in their book, they could just do as Tolkien and sing their songs[^2], in public. Taking that one step further would be to establish rapport with readers by teaching them how to play those songs, e.g. in a piano or a guitar. It's like going back in time, to when the Iliad was sung and performed instead of written. Though there's no need to negate technology; if a writer is sufficiently technologically inclined, they could forego the linear story and instead release interactive fiction for musical instruments...Hm, I'll see myself out now.

[^2]: https://tolkiengateway.net/wiki/Category:Songs

Hmm, those are some worrying news for Zephyr, but also for Linux. For reasons that escape my full understanding, a lot of embedded hardware needs Linux to work. If Zephyr, which I bet has been carefully crafted for microcontrollers, leaves some unnecessary things on, who knows what even an extremely trimmed down Linux running in, say, the beagle board, gets away with.

I loved the video. Also, I had a lot of fun watching it while imagining I was living in 2051 and this whole progression had ran its course. It's impossible to predict the future, but certain events can negate whole branches of possibility. This one video in particular prunes the "we never figured out robotic motion and thus all the science fiction scenarios built on that premise never came to happen" subtree.

It's a movie for theaters and not a flashback edgy post; the success of the work is about pleasing the audience. That is already a thing in Homer's original Odyssey, at least as I read it in high school. But now with a few more decades on my ribs, I can quickly find out that the text that I read came from oral traditions that spanned from 1200 to 600 BCE, and after finding its way to written form, a cultural chain of transliterations:

Archaic Greece -> Hellenistic Alexandria[^1] -> Roman Egypt -> Byzantine Constantinople -> Renaissance Italy

Each age was obsessed with something, and the latest stages in that chain are heavily marked by Christian values. So, yeah, swords and explosion and stuff and don't do anything we would consider obscene today which includes sex, though movie directors are completely unabashed about bloody horror. To me, it's not a reflection on the director, but on his audience and their society.

[^1]: https://www.getty.edu/art/collection/object/103T4T

A bit tangential, and in the topic of nostalgia for old ways, remember when every block had a photo shop? Couldn’t we get a “PCB shop” in our bright near future? I wonder if the density of enthusiasts that need PCBs made is so low that the entire market is captured by only one or two companies based off China subject to the realities of shipping times.

There's often an issue of perspective. Imagine you visit an 11th century monk in the occupation of copying manuscripts by candle light. This is a guy that spends long cold winters hunched over parchment, using sticks with god knows what passes for ink to copy and illuminate beautiful texts. You tell him that you will buy manuscripts from him, as long as they are typeset in Times New Roman regular, 12pt, and please use this laptop for that. Also you are setting shop with all sort of modern niceties next to the convent, including heating fans and desk electric lamps. Of course the monk will be heart-broken and incredibly grumpy at the loss of time to practice the craft he loves, which is writing and illuminating by hand. But those heating fans are something alright.

We already have desk lamps and heating fans, and many other things which make daily life convenient. But if you hand me a piece of paper (or better, a notebook), I'll fill it just enumerating the amount of diseases I want gone from the world. At least in theory, we know the laws of physics and how things are put together, and we know many of the impossibles of today are simply due to extreme scarcity and cost of qualified intellectual labor.

Maybe what we need is a shift from extreme pride on a specific set of technical skills, to priding ourselves in understanding enough of the world to solve some of its problems.

Interesting that you bring daydreaming to the conversation. I've got an issue with it, and it's that I get interesting ideas then. But nothing comes out unless I sit in front of a computer to do anything with them. And computers have all the distractions.

I have had, and continue having, way above average use for my handwriting. It works for all sorts of math, quick diagrams on anything from geometric reasoning to electronics, composing tricky fragments of text, taking notes on something I'm doing that requires a myriad of easy-to-forget details, and so on.

But my 1st grade teacher and my 1st to 12st year classes variously agreed on how atrocious my handwriting was. Famously, my 1st grade teacher told my parents, "there's no hope". She was right. And, four decades later, it irks me to no end that every time I grab a pen and start writing, the ghosts come to look over my shoulder and judge. "Look at it, he's at it again. That 'f' looks like anything but... That's not a proper uppercase 'I'... you could summon a demon with it, but it would be a pitiful one. That 'g' is vicious. Is that a '1', a '3' or a '7'? Didn't we fail you enough times in basic arithmetic for that? Also, that line of text is sagging as much as your sitting, boy..."

It doesn't make me happy to say it, but the American LLM companies were first. Capital in the rest of the world is way more conservative, and I can't imagine the mega-investments OpenAI and Anthropic managed to secure happening anywhere else without existing proof that "thing is profitable".

This. Forgetting is the biggest crime. Anything in the other direction is atoning.

Well if AI destroys bad music videos by making them cost $25, I say it's a win. What will be left will be seriously scripted productions.

Another day, another model and another pelican :-)

I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?

PCBs are very neat, but that shipping time is a boomer when you are prototyping and you really don’t know what you are doing.

Honestly, I don't take offense at the joke. There's a reason freedom of expression has some degree of protection in many countries, and it is so that people can slander each other's religion or lack of it with impunity. I mean, I'm pretty sure it wouldn't be constitutionally protected if it were about football clubs' rivalries. So I slander God at every opportunity I get, while actually being an atheist, because I find the given solutions to the problem of evil abominable and have mild objections to the morality of whoever swallows said solutions. Though I would never bar people of faith from using an operating system I contribute to; why in hell would I deprive them of a platform to read my slander?

A few years ago I bought a SUP and there was a motto on it "life is easy: just add water". I've done enough with that SUP to learn that the motto is actually true. I can imagine some people take it to the "life is great: just add an ocean" extreme.

Deeply disappointed to read it doesn't work for iconoclasts.

Up to five years ago[^1], we lived an age of pure speculation about intelligence and minds. That age brought about multiple theories of the mind, and to me, their finest legacy are arguments of the form "$X is not $concept because $theory", where any given day $X can be "LLM" or "LLM+harness+tool", "agent" and $concept can be "intelligent", "aware", "rational", "long-termed", , etc. Those arguments, in the context of $theory, are not wrong, and they are in fact quite useful. But there's only so much utility in a sieve.

We are going to need positive theories of the mind, and we will have to build them using terms and bits that avoid confrontation with the old guard. So, maybe instead of "mind" we will say "artilect"[^2][^3], and then move on to engineer desirable properties. For some of those properties ("alignment", "paucity", Kantian analysis and synthesis) we are already there or aiming for, but as we design to take on more and more challenging deeds, we may need to go back, steal and rebrand other old ideas. For example, "long term goals" and "self-preservation" are strongly linked in humans. Can we reify that relationship in the new artilects we make?

Other things we will need to make from whole cloth. For example, minds with an intuitive understanding of, and capability to manipulate: quantum mechanics, biological organisms, the workings of an ecosystem, the stock market, a stock company, or itself and other minds.

[^1]: YMMV, depending on where you draw the line between fine intelligence and stochastic parrots.

[^2]: Not the dictionary meaning, but the term as used by Alastair Reynolds in "Blue Remembered Earth (2012)"

[^3]: We may not use the term "LLM", because of the stochastic parrot crowd, because LLMs become one of many ingredients, or simply (but unlikely) because we find a completely different tool for the kit and discard LLMs entirely.

So, we are giving up privacy in all our electronic communications because a bunch of companies can’t stop running algorithms to drive up engagement? From the article above, it seems Snapchat messaging would work just fine if kids had to touch phones to exchange contacts. Versus putting everybody in 24x7 AI surveillance, the lesser bad here is to forbid those algorithms, even if that bankrupts the platforms.

> I keep telling people about such things and I am looked at as nerdy, geeky or boring.

Hm, yeah, we have our work cut out for ourselves. Politicians can't do nerdy nor geeky, but it's their job to talk in a way that moves people. That's why we keep electing absolute idiots that can't even speak that well, all things considered, but who can "charm", for a given definition of charm of course. To be heard we need to remain nerdy and geeky at our core, but talk in a way that moves people.

In this concrete instance, what I do when somebody brings Chat Control to the conversation and other listeners start to roll eyes, is to derail the conversation with colorful yarns about how we did surveillance in the old days of the Soviet Union, and what we did with anybody who was rattled for giving a foul mouth to the Party. "Yes, we didn't have Siberia, but the heat and the savage ants in those sugar cane plantations were damn fine, and honestly you don't need any particular geography for a good old beating... Catching them dissidents was the hard thing, but it all would be so much easier these days... Hey, have you noticed how you talk about one thing and Facebook start popping ads about it almost at once? Does it listen to all our diatribes? I'm pretty sure that's the stuff Chat Control wants..."

What are the updates for, I wonder?

Video games, like all forms of art, are about stirring emotions. I don't own, nor have never owned an Xbox. But when I think of the device the first thing that comes to mind is "Microsoft" and "Windows". So I consider all the little beancounters and MBAs at Microsoft who are always optimizing for profit and that have made Windows nightmarish, and I imagine them with access to an "emotions machine" they can manipulate to make number go up, and can't help but think that the device is a pocket dimension of hell but more or less useless otherwise.

All said, in order for Microsoft to fix XBox, they will also need to fix the Windows desktop experience. Otherwise people will think "ouch, I don't want to buy a cousin of that creature. Better go for a Nintendo or Steam Deck or something..."

I can't say anything about the "centuries" part, but the rest sort of checks. In those old soviet countries there was no such thing as "customer service", the politburo didn't get around inventing it and every economic cog was created from above. If a modern American had seen how things were done there, they would have wrongly assumed that a powerful pulp-and-paper lobby was in control. Also, when the thing in question was made in the West (which was often the case for high tech stuff), and somehow smuggled under the iron curtain and Western sanctions, customer support was unaffordable or simply out of reach.

I wouldn't mind reading that and having a good chuckle while processing an MR, as long as the comment had been crafted by a person. But now I think writing in grunts is gonna become the thing. "pete? you good boy? we won't hire you/ paper you gave HR girl with older gigs? remember it pete my boy? too many words/ words were too long/ dots you used dots/ you scratched long dash with knife, but baby saw scratch/we don't ai pete/ask if they have job in next cave/good luck."

> but after 5-10 generations of a combination of artificial and natural conception you could end up with meaningful loss of fitness

Yes, if we end up in some corner-case dystopia where evolution and natural selection continue to be in charge of fitness. But evolution and natural selection bring much suffering to the unlucky. In other words, if you go to a hospital, you'll quickly learn there's far more human suffering caused by God and Nature than by the "cruelty of man". Though common sense is never assured victory, I look forward to a world where our children live healthier and longer lives due to us properly messing with God and Nature.

Slightly tangential on Swedish society, there are similarities between USA and Sweden. There's a large segment of society that is white and very blond, and there's a largish segment which is not. Along that same line there are all kinds of divisions: economic, education, religious, sets of values, and of access to things and possibilities. What pisses me off is that the cast of "CEOs of successful companies" live in an sphere of privilege where they really are not bothered at all by the brown people. They in fact have plenty of places to go, a vast archipelago, out of reach for anybody who can't afford a boat. Though they get all the benefits, including cheap qualified labor from people who had to leave their homelands displaced by poverty, conflict and war. I'll switch VPN provider too.

One of these days we will elect somebody who is corrupt and morally corrupt, incompetent and poorly educated and who'll promise to screw us over many times and in many positions, and we will let him just do so so that there are concentration camps for the brown people.

> There's something incredibly peaceful about being in the hands of an expert you trust. You don't have to worry anymore and can let them guide you through the process.

> AI can absolutely shatter that feeling in an uncomfortable way...

As a mental experiment, think how things would look like if the AIs were right more often than doctors. Then living longer and being healthier would imply living with a lot of "technical medical worries" that you don't longer get to outsource to a black-boxy human wearing whites. I know some people who are already ridding that train.