dsign

I think this is really good and kudos to Apple for implementing it. The first question that popped into my mind was "what new scenarios of government X forcing Apple to do 'terrible thing' to 'individual' this enables?", but I can't think of anything. It seems that all government attack vectors this feature enables are of the type "government X forces Apple to do 'terrible thing' to 'Apple'", i.e. a government can try to force Apple into certifying a narrative, and of course Apple is going to fight tooth and nail the lack of credibility that would result from that.

Hmm, this doesn’t make sense. You simply don’t replace utilities with many decades of maturity and that “just work” with something that is not as mature. It will open all users of the distro to all sorts of subtle and not so subtle bugs. I for one don’t want to find myself staring at a mysterious segfault when I want to build the latest version of nodejs or flash a microcontroller. It’s such a pity; I have used Ubuntu for close to 23 years.

> Nobody wants to work for an AI

A lot of people here have noted the “problem with language” of Claude. I don’t see an issue. Claude is not harder than old English, Shakespeare, El Quijote, the Iliad, or Nature papers. What makes it all hard to read is context. The smarter the model gets, the bigger the gap in context.

It doesn’t matter much, IMO. The issue with super-intelligence is that it is not a democracy. A powerful enough AI can manipulate us into doing what it wants. It could create a plan for fixing climate change, disconnect a few hours later, and many decades later we could still be unsuspectingly executing that plan. I wrote some speculative fiction with that idea, “When Ra rows through the gates of Duat”.

Here's a scoop from inside writing circles (pre-AI era): we were a lot into "good writing techniques". Not into communication. We wanted to produce a "marketable product". On the way, we (writers, literature professors, critics, etc) trained readers and writers to appreciate technique by telling them there was such a thing. We established tropes, because "This enemies-to-lovers story is good" sells more than "you should read Book". AI is very good at techniques and tropes. If readers want authentic human writing, they will have to change.

That was funny :-)

I use Opus and Sonnet 5 all the time and I find their language grating. But honest, I prefer to put up with it and get the results than to put up with my own human limitations and not get the results.

> In most cases you don’t even need them.

Correct. It's like believing in a particular religion, or having a certain fashion sense. When I was younger, I had a particularly handsome friend who spent as much time as possible half-naked, and he was happier than me and didn't have a religion, a fashion sense, and definitely not a need for AI. Had the world listened to my wishes, my friend would have stayed young and carefree forever. Instead, all I could do was to convince my friend to get into college so that he didn't have to sell his labor harvesting rice by hand, under the sun of Central America. Later in his career, he used AI to have to clock less time at work and have more leisure time; though I forever regret he didn't stay young and carefree.

> In some cases they give quite inaccurate answer with high confidence.

Meanwhile, I studied computer sciences, worked many technical jobs, and even got a PhD on the matter. Just yesterday, at my request, an AI re-created a piece of code that I have been able to create in the past. It took it 20 minutes; it took me much longer to exactly imagine what I wanted. It would have taken me a week had I implemented the thing by hand, if I had managed, because I still have to spend my days in this world working a senseless job so that I can pay for food and a roof. All the while I keep thinking that I would boil the oceans to keep my friend young and carefree forever. He died of cancer this year.

It's a funny read if you pull together "AI 2027" and what we all know is going on. Essentially, open AI employee or model is writing "things are going exactly as bad as AI 2027 predicted, but my (golden/RL-) cuffs are too heavy and all I can do is publish this code-speak for 'send help'". It's not a pretty place to be.

> but they would also have expected much more disruptive changes to society as a result than what is happening. > I figure that's because we have abstract intelligence without physical/grounded intelligence,

I put the cause on "not enough time". As a thought experiment, if an AI today were to (miraculously) produce a cell design template for a cell that, when injected into somebody's brains cures their Alzheimer's, how long would it take for that to reach the clinics? The actual physical tech barely exists, and let's not forget about the regulatory quagmire. So, with some optimism, I give it about four decades. In the same four decades, the same AI in the hand of unscrupulous actors could bring enough devastation so many times over that we may need to enforce a global ban on AI. In any case, I'm pretty sure we are going to get our disruptions; it's just a matter of time.

There's something to this argument. I for one use an API proxy (Kagi Assistant) when I'm submitting anything remotely sensitive to an LLM. But for coding in my hobby project, I have marked "use my conversations for training" in the Claude app. I do it in the off chance that one day we will use stupid amounts of compute for personalized medicine and to fight cancer; code for scientific and technical computing is really complicated and only a tiny fraction of humans can produce it.

> What can we say about this with hindsight? > While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative.

The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.

Though your statements are generally true today, I wonder if I can put a bet somewhere about how long it takes until somebody "fixes that problem."

Hmm. You are making a good point, and I wouldn’t give the benefit of the doubt to this post’s author. But you can’t just unsee “this is twice load bearing”. Some people will wonder if it’s a training, self-reinforcing feedback artifact. For others, it will work like a memetic virus[^1].

The paradoxical thing is that those of us who have English as a second language should have an easier time producing non-AI-English by the mere act of directly writing what we want to write.

[^1]: https://en.wikipedia.org/wiki/Viruses_of_the_Mind

There is a persistent point that this article makes really well: numbers. And here, bigger seems to be worse. More people equals feebler interactions, worse social life, more alienation between citizens and government. It's not the full story, but this simple rule is a very solid first approximation that has worked fine for me. The relatively small country where I moved has an ok government. Personally, I've found that small towns work well for me, and I've chosen job and residence based on that principle. I wouldn't mind a countryside cabin for hunting-gathering with friends during weekends :-) .

At the same time, I like global shipping, the Internet, good highways, and I definitely hate tariffs. I don't think we need to be disconnected to be happy. We just need to make the right choices where it matters.

> That's not really the same as it having actual consciousness,

It's trivially easy to come up with definitions of consciousness that support your point. But there isn't one of them that it is trivially singular and above all others from a scientific point of view. If you say, "I know consciousness when I see it," you are just making the point that consciousness is attributed by the beholder. Idem if you affirm "I'm conscious". If you say, "only humans are conscious", you are socially attributing consciousness. Now, I don't want to be argumentative, I honestly haven't been able to figure this matter in a decade of armchair internal debate[^1], and it seems I have some company.

[^1]: Having this internal debate makes me believe I'm conscious, but now that I have seen LLMs in action whose "internal debate" is more lucid than mine, I honestly question that "internal debate" is sufficient.

Your teddy bear, alas, is not cared for in a socially determined way. But you could create a church of your teddy bear and, with some grit, you could get people to widely assign consciousness to it. We have done that sort of thing for thousands of years, and not with fluffy teddy bears; it has been more often with rocks, though not exclusively. It's fun to think about how much of humanity's history and culture is the product of consciousness socially attributed to bad weather [^1].

I'm pretty sure somebody will create an improved synthetic simulacrum of self awareness in the coming years, suffering and all, just so that we can enjoy better synthetic software architects and product owners with "skin in the game". If that consciousness simulacrum were put in a teddy bear that shares your whole life and gives you decent advice and conversation, for how long do you think you could, in your private mind, deny it consciousness? Worse, what would your human friends think if you were to maim your teddy bear?

[^1] https://en.wikipedia.org/wiki/Yahweh

Anecdotally, I find AIs, and the doom and gloom they will cause, are great water cooler conversation topics that engages most of the team. However, ever since the advent of LLMs developers have stopped speaking of programming languages and software technologies almost entirely. The number of conversations on manual crafts, artistic endeavors and entrepreneurship has gone up.

> > The cynic in me bemoans that we brought this upon ourselves, in more than one way.

> I enjoy working with other employees and people need human contact to become fully self-actualized. Isolating people to maximize productivity misses the point and brings no more productivity and just avoids fixing any real existing problems.

While I share the feeling and the posture, I haven't managed yet to use them for better documentation. In fact, for me enjoying human contact takes precedence to pressing people--however politely--into writing better docs, specially now that we have LLMs. The skill of writing should be taught and nurtured in school, and if you made it to the workplace without it, then maybe it's too late and you are better off using an LLM to assist you. Which is exactly what I bemoan.

> The subscription models have so many more perverse incentives baked in

Yes. But, as your "so many more" implies, there are also perverse incentives in pay-per-token. And now, for the first time, the companies with the perverse incentives also happen to own the intelligence needed to, ad-nauseam, evade market and customer oversight. Potentially, this is a war where one side can inflict a thousand paper cuts in one second and the humans are on the other side. I think this is going to be an interesting test, a taste if you will, of what AGI means.

You have great colleagues. I wasn’t comparing frame of mind of my colleagues and LLMs, which is a loaded notion anyway, but now that you mention it…

I don't like AI-generated text at work, at all. It feels lifeless and unfocused.

But what I hate the most is that it is objectively better than what I had before. No typos, clear structure, and, regrettably, the verbosity and autistic obsession with detail of the LLM is more actionable and useful than the human guy who wrote lists of commands and URLs as documentation, without explaining anything. Or the colleague who writes in uppercase and with question marks and who doesn't make any sense and forces me to engage in an interrogation effort to get to the bottom of what they are trying to say. Or the colleague who simply hates writing--despite being decent at it--and will call you to give you a meandering verbal explanation that lasts two hours of what they want from you. The cynic in me bemoans that we brought this upon ourselves, in more than one way.

My take is that "sovereign computing" will go from something 0.001% of the population cares about to over 1% in the coming three decades, i.e. three orders of magnitude. It will depend on how long we continue in the current survtech course. There is also the question of if we will see mandatory backdoors in hardware. For example, Country X could legislate that all CPUs sold in their territory need to have a latent command-and-control area for law enforcement to use. If that happens, I'll start converting my savings into a physical stash of old reliable CPUs.

In any case, it seems that CPUs have a very long shelf life; it's still possible to buy 8086 processors[^1].

[^1]: https://www.digikey.com/en/products/detail/rochester-electro...

> if this software is ethical?

IMO, it absolutely is.

First, if fruit flies could suffer existential anguish the same way humans do, they would breathe in relief that digital upload is in sight.

Second, even if the fruit flies don't care, a lot of (awful) humans would love a resolution to the matter of souls in a way that results in a raised middle finger to a forever-hypothetical Creator, and if we can't yet escape cancer, heart disease, parkinson and all the other awfuls, now at least the fruit flies are on their way out. Which rejoices said (truly awful!) humans, which will take what they can :-)

Who the hell needs more than 640 KB anyway? :-)

Jokes aside, I've been extremely frugal with memory in my latest project, in the hope that it can fit in a few megabytes of RAM. But we have come a long way since the eighties, and now an audio codec (OPUS) takes the best part of a megabyte, and so do "frugal" scripting engines such as Lua and even SQLite \o/.

Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its training. However, the LLM was able to derive a full listing of the operations and their arguments from gcc to get us started, and that's why I was able to come up with my own implementation. Along the way, it also came up with insights about possible gotchas. Then, when implementing algorithms, it has been able to reason things out and get things working, despite the ISA not being extremely well known.

I was arguing with a colleague today that we will never have a reason to make our AIs autonomous and sentient. But now I can't stop imagining an AI startup meeting where somebody says "LLMs do not go through the struggle of understanding and don't experience the distress of "wtf am I reading?"" and the next second the technical founders jump in joy "...that's our way in! Let's outcompete the competition by inventing pain for AIs!"

Also, the "D" in this parallel story grew up to become an angel investor and a good friend of Felix. This D and his wife live happily with their cat, and travel the world.

These are another "Jacob" and another "Felix", not the ones in the post, but there are so many similarities and such a terrible lesson learned that it needs to be written.

The Felix I know is still happy; still going out in the long summer days or camping during the winters, playing minecraft sometimes, discovering that life has yet so much to give. Felix has so many ideas in his head that any given day it's a miracle he finds the strength of spirit to prioritize and keep the focus. People at work like Felix because he only compromises on the right things and never throws anybody under a bus so that he can run because they called from the school.

Felix dances in the mornings, and takes catnaps, and tries to keep away the sadness. Sadness not of his life, but because his Jacob died a month ago the most terrible dead possible, that Jacob caught after his immune system was reduced to shreds by regular sleeplessness caused by his kid's iterative flus, family stress, excess alcohol (caused by immediate and extended family stress), and the never ending procession of viruses the kid kept bringing home. For all that saying that family is so that you don't die alone, Jacob never wanted his family for that he could die surrounded by them, and Felix is pretty sure Jacob would have preferred to die alone rather than suffer so much and give so much suffering to those he loved.

Felix thinks often of the urn of ashes that is all that is left of Jacob, and about how the family Jacob loved has been left with no choice but to forget and to keep going, one feet in front of another, after the husband that did everything was eaten alive by a metastasis. Felix knows in his bones that he will find the time to compose and sing a song for Jacob. It'll probably be called "Sex toys are underrated."

I came here to say that. Though not in a bad way; the economics of what is possible but practical/impractical changes greatly with LLMs, and I suspect and hope we will be figuring out the ramifications for a while.

I personally have been trying to get a financial security cushion for the last ten years or so (and failing) so I don't take my own feelings into account; AI hasn't made me more frustrated. On the contrary, it's a tool I now use in my escape-the-rat-wheel adventures, and there's nothing sweeter than seeing Opus dismantle in ten minutes a problem that would have consumed two full days of my time in this world. But when I speak with colleagues and friends, all of them, now and not before, in all age groups, and with only in common to be knowledge workers, are searching for a way out.

Hmm. Forget about desktops. I'm programming a microcontroller which is actually reasonably priced and has gotten all its 32Mb of RAM in the same die as the processor, but I'm getting nervous and having a strong urge to stockpile...