kragen

Over the last few weeks, I've been eliciting a compiler for an experimental programming language called 5emel from Claude Opus 5.5. To give you the flavor, here's a randomly selected function from its compiler:

    # The table of the program's own functions, by their descriptors, which
    # the runtime searches for --memo-every and --memo-all.
    def function_data(:defs) =
      :rows = loop :next(:r = defs, :acc = []) (
        | [] = r; acc
        | [Def { :fname, _, _, _ }, ...:more] = r;
          next(more, [acc, "  .quad d_", fname, "\n"])
      );
      ["  .balign 8\n  .globl rt_functions, rt_nfunctions\n",
       "rt_nfunctions: .quad ", show(length(defs)), "\n",
       "rt_functions:\n", rows]

There's a lot of code in 5emel I haven't read; there's something like 15000 lines of TypeScript and JavaScript and 3000 lines of C. I've read almost none of any of that. I haven't even read all of the compiler, which is only about 1600 lines of code.

Despite that, I think I've been putting my extensive past reading of programming books to pretty good use in this process. 5emel is directed at finding a workable, though not optimal, general solution to one of the famously most difficult problems in computer science, and if it is successful, I think it'll be a real advance — and one that LLMs can use as well as humans. Better, maybe.

Craven?

As I pointed out above, Anthropic places a lot of requirements on its users already, but if Claude Code TIP were optional, presumably, savvy users would turn it on voluntarily, unless they wanted to use Auto Mode. As Anthropic's study clearly documents, the current manual approval workflow basically doesn't work at all, because the users just approve 87% of dangerous actions. So there's no reason to use it, except maybe very rarely.

> one conclusion they/some of the scientists drew was that the culture of the Nazi regime - the paranoia, fear, and violence - made a project on that scale impossible

This makes me think about the pervasive classification and paranoia in the "defense" contractors supporting the US Department of "Defense". Any information that's classified, even at the lowest "confidential" level, is only provided on a "need-to-know" basis. And unclassified projects are sort of looked down on, so there's a perverse incentive to classify more than necessary. This "secretive, isolationist approach" seems like it probably prevents a lot of groups from "sharing information or resources", in the article's words (though it concludes that wasn't the biggest problem facing the German effort).

When sharing classified information that someone might not need to know might be a crime, and might get your clearance pulled even if it's not, the incentives run strongly counter to sharing information. Consequently, I confidently claim that there are very, very few people in a position to serendipitously make significant discoveries that involve combining two or three different classified facts.

(I don't actually know this, because I don't know any classified facts.)

Maybe this is part of the sclerosis that causes the US defense-industrial complex to "basically produce useless junk", in the words of Les Earnest (https://www.youtube.com/watch?v=Ych6pvLMKCU), and which prompts Sergey Tovkach to say, "There isn’t a single Western drone that’s proven itself effective in this conflict." (https://www.youtube.com/watch?v=RmfNUM2CbbM). Maybe that's why Russian propaganda in 02024 reported that Ukraine had to withdraw the USA-made M1A1 tanks far from the front (https://www.youtube.com/watch?v=aCIN-t3VxJ0).

Not only is it a serious proposal, it's been used in practice in every airport X-ray baggage scanner for decades, where it's called Threat Image Projection (TIP): https://www.xscann.com/products-HiTipPlus.html and it's actually a regulatory requirement in that context in a number of countries: https://caainternational.com/course/threat-image-projection-...

It's AI slop. If what it says is actually true, hopefully a human will post it.

Maybe you can include some percentage of intentionally malicious commands, but not actually run them if the user approves them. Just blare an alarm telling the user they misclassified a malicious command, and maybe pause the Claude Code session for a frustrating 5–15 seconds. Video games can elicit truly astonishing levels of performance from human brains by using this approach. I'm guessing it's probably outside your Overton window, which is too bad, because it has the potential to work a lot better than the 89% you get with "Auto mode", which is far too low if anything really significant is at risk. That's not even one single order of magnitude lower risk than just "allow everything always".

Speaking of really significant things being at risk, it's pretty infuriating that the web-chat Incognito mode can't write and test code locally, and neither the non-Incognito web-chat nor the local Claude Code client can be shielded from the knowledge of my email address and my geolocation. (Even the Incognito mode knows my geolocation.) This gives users no alternative but to put privacy-sensitive information at the disposal of Claude Code, which is far too unreliable to be entrusted with it. It's tempting to attribute this to Anthropic's corporate DNA valuing users' interests at a very low level, but conceivably it's just an oversight, and if I mention it here, it will be corrected in a month or two?

Currently, my assessment is that Claude Code, and in particular the Opus 5.5 model, is head and shoulders above the competition in software development capabilities. Nothing else is even close. But I've seen several small signs like this which suggest that Anthropic, as a company, is maliciously misaligned. So, if Grok, Qwen, GLM, or Astra starts to come anywhere close to being competitive, I'll switch as soon as I can. I imagine that most of your users feel the same way.

Humans gotta hume.

I think it might have worked better if they had given it a name.

The MHz ratio really understates the difference.

66MHz probably wasn't an 80486DX; as far as I can recall, those only went up to 50MHz. It would have to be a 486DX2, with the RAM running at only 33MHz. https://netlib.org/performance/html/dhrystone.data.col0.html says that a 486DX2/66 was about 31 to 51 Dhrystone MIPS.

https://www.riscvschool.com/2023/03/09/t-head-xuantie-c906-r... says the T-Head (Honey Badger) C906 RV64 core is about 2.3 Dhrystone MIPS ("DMIPS") per MHz, so we should expect the 480MHz C906 to be about 1100 DMIPS.

So the clock speed is only ≈8× higher, but the actual speed gap is probably closer to 32×.

And, for vectorizable workloads, we should expect the V extension to give us another 4–8× boost, so 128×–256× the DX2/66, which didn't even have MMX.

I wouldn't expect them to. Let us know if you find otherwise!

Gizmodo's article did suggest banning knives from the homes of stupid people, Turkish or otherwise. Sadly, I don't think this degree of insanity is that unusual anywhere.

Even on a single-word basis, "male" might be make, makes, Maria, Maia, narrow, Maddow, marrow, Mariya, Marita, mafia, males, mallow, Maslow, male's, or masks. "Chauvinist" might be coatings, costing, coaxing, coasting, crating, contingency, contingent, contingencies, chauvinistic, Chauvin, or chaining. "Pig" might be PC, PIC, PCC, pox, poof, PVC, pH, put, pug, PV, Pooh, log, or pub. These are all possibilities that were actually presented when I tried to swipe these words on my Android phone, twice each. Taking into account additional context will broaden this set of possibly confusable words, not narrow it.

We might not talk about a "narrow contingency put" or "Maria crating PVC" very often, but an uncensored autocorrect would almost certainly correct those phrases to the far more common "male chauvinist pig" in the rare cases where someone did try to swipe them.

I'm guessing that somebody tried to swipe "Halas" or "Hamid" and it came out "Hamas", or "Tibetan obedience" and ended up with "Tibetan independence", which was followed by a visit from the police or being blacklisted from PayPal. Or "I want to question your artsy friend about the gallery" came out as "I want to question your sexuality, friend about the gallery".

There are cases like https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl... where, in 02008, Ramazan Çalçoban texted his estranged wife in Turkish, "you change the topic every time you run out of arguments", but the lack of the Turkish dotless "i" on his cellphone turned "sıkısınca" into "sikisinca", which looks more like a misspelling of "sikisince". So he ended up texting her, "You change the topic every time they are fucking you."

Long story short, his father-in-law murdered him for thus insulting his daughter's fidelity, following which he murdered the daughter in question, and the father-in-law committed suicide in prison.

So there might be certain words and phrases that it's best for your computers to not send to other people on your behalf unless they're really sure you mean them.

It's also probably the correct approach to take with an awesome power whose true danger is unproven.

So, maybe the faked text in the post is documenting a real visit because maybe he didn't fake the photos, only the text?

No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.

It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'

This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.

Maybe he didn't do any original reporting, but just prompted Claude to write a science fiction story where a reporter visits some fictional factories in China and blogs about it?

Every time the article said, "I did such and such a thing," I thought, "No, Claude, you did not do that. You don't have a body. You're making this up."

That's plausible, although I don't think it's actually correct. But there's a pretty extensive set of AI luminaries who are concerned who can't be dismissed so easily.

Maybe coreboot is what you are looking for? Or DeviceTree? Or Open Firmware?

Or something for a specific board. UEFI or BIOS is just a compatibility layer, and its only advantage over saner ones is that it's already installed and working on certain cheap, fast hardware you can easily buy.

But maybe that's less of an advantage now than it was 28 years ago.

To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.

My demographic claim? It's obviously correct, as anyone with the most casual acquaintance with the question knows. The worried people, like Geoffrey Hinton, Bill Gates, and Dario Amodei, might be wrong, but they certainly aren't ignorant outsiders. (Neither is Yann LeCun, of course, which is why I mentioned him: he's an important exception to the rule.)

How would touching grass help? Do you think the grass is better informed about AI risk than you are?

...that's plausible, actually.

Probably a site called "Hacker News" is not the smartest place for you to be implicitly asserting that thinking about things is bad, or that those nerds who do it are out-of-touch losers. Maybe you want to be on Rugby News instead.

On Hacker News, by contrast, what is valued is thoughtful discussion grounded in the kind of rigorous theoretical knowledge that comes from extended concentration, usually but not always indoors, where it won't rain on the computers.

The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.

> probably destroy almost anyone who survived the initial blast and fallout.

The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.

> Can you explain how exactly an AI would develop a novel planet-killing weapon

I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.

> I seriously don’t understand this argument

That is a point that I think everyone can agree on: you do not.

Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.

I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.

It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.

As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".

Oh, if you aren't him, my apologies.

Yes, build-time.

Coincidentally, I had Claude fix 18 bugs in PCC earlier this week actually. But PCC itself doesn't support the UEFI calling convention or PE32+ image format. Those can be provided by something much smaller and simpler than LLVM, of course, but I'm not there yet.

HPFS2 looks appealing. Did you implement it? And what were the design blunders in LEAN you decided not to go into in the footnote?

Given that normally UEFI only needs to read the filesystem, not write it, I think a more appropriate filesystem might be something like ZIP.

Yes, I think that's a fair assessment, although the fractal of stupid runs deep in FAT32: a separate 5-bit field for seconds within a minute? Separate fixed-width fields for filename and extension? A 4 GiB file size limit?

But, although FAT32 does indeed have the virtue of relative simplicity, that's no excuse for VFAT.