meowface

The reality is both are true. The code artifact on the disk and the idea object in the brain should both be treated as contingent, relative, and incomplete, and unvertain.

Prolonged conversations with humans and AIs about both ideas and code is very useful.

Good post. I like this example:

>Say you want to draw a perfect picture of an owl. You have this beautiful image of it in your mind. But you can’t draw very well, and in this analogy the drawing is the code. So what comes out is a stick-figure owl that’s barely recognizable as an owl.

>In the code-is-the-source-of-truth version, that stick figure is now the authoritative artifact. We analyze it, document it, take notes on its proportions, and spend a whole bunch of time iterating on it. We’re running in circles around a bad drawing.

My only disagreement with the post is over whether articulating is is the main bottleneck. Right now, there are still issues and oversights humans would tend not to make even if you have model context and documentation accurately and clearly capturing your full intent.

Each year will decrease the rate of this, but software engineering will probably need until maybe 2030 until we've crossed a threshold towards solely being product managers/designers. This advice still applies today, but you do need more wrangling on top.

Thiel, unlike many of these other figures, is genuinely insightful. He also has one of the best investment track records. And he knew to back Trump in 2016, not back him in 2020, back him in 2024. He's one of the rare "right-wing intellectuals" who's a genuine intellectual.

The core issue with him, in my view, is his openly anti-democratic views.

He identifies as a libertarian, yet finds democracy loathsome and either wants a monarch or finds it strategically useful to fund and support initiatives that increase the chance of there being increasingly monarchical leaders in places like the US. I don't think that is a moral ideology, and I also think it is generally not a good one for achieving positive outcomes for a majority of people.

There are genuine critiques of democracy and its tendency towards shallow populism ruling everything, but, as usual, the meme defense is the correct defense; it's better than alternatives, like permanent "rule by 'CEO-monarch'".

I bite the bullet on this one by saying we should in fact do something about that. All of it. And we will, when we can. Probably before 2400.

It should be a tiered system, IMO.

1) Where possible, don't test in animals.

2) When testing in animals, first ensure that there is reasonable confidence the risk and severity of harm from the testing will be minimal. Only begin initial in vivo testing after this has first been achieved.

3) For test animals that are or aren't harmed, allow them to live happy, (relatively) normal, natural lives. Don't end their life early unnecessarily, with exceptions for humane euthanasia if the testing has unexpectedly caused significant, persistent suffering. Treat them as a pet owner would treat their beloved pet rat.

Some may consider it extreme, but, in my view, in an ideal world, people would treat testing in animals almost like testing in humans. The researcher should first ask "would I feel comfortable testing this in a voluntary human test subject?" Non-human animals would still be the first line of testing, since harm/death in a rat/mouse/chimpanzee is less morally bad than in a human, but I think this should be the mindset.

Hmm, but is it not the case a much larger model could potentially reason more in latent space, depending on how it's set up?

I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.

Trustworthiness arguments aside:

You're saying you think there might be machine demigods in 50 years, but that one shouldn't use the pace of AI advances today to update positively on the probability of that?

For agentic coding, Claude Code with the latest Fable or Opus is still just better. That's what people care most about.

Unless Gemini CLI greatly improves and Argon works well in it, this probably won't make a dent in the coding market.

Let's check in in 50 years and see which of us was right.

Also check out this thread: https://x.com/tszzl/status/2109822082024656927

I trust him more after those two statements, yes. One would need to be an especially ridiculous populist to misinterpret those statements in a way where one would place less trust in him as a result.

I did negatively update based on reports of colleagues saying he has a tendency to be deceptive, though.

The current pace of advancement and the fact that fresh writing data will likely not be a requirement at all to continue accelerating. The pretraining paradigm no longer needs this.

It's not a certainty by any means, but to act flabbergasted at the thought of the existence of ASI by 2070, in 2026, on a tech site, is just silly.

That likely would not be exactly how it plays out. It would be a rogue or misaligned AI that the companies completely lose control of or that a different actor evilly misuses.

But, yes, there should probably be laws that make it so if you summon a demon that causes a mass casualty event you face criminal penalties.

There are some confounders there when it comes to Google.

I am not saying I trust them not to make a mistake that leads to catastrophe. OpenAI in particular is not doing a great job. Anthropic or OpenAI may in fact indirectly kill millions or billions of people. I still think they're probably less clownish than most possible alternatives.

I don't think anyone or anything can really be trusted with this. We need extremely strict international and national regulations, before things really start accelerating (we are still in the very very early stage). I still just don't really feel any specific disdain for OpenAI or Anthropic (particularly Anthropic).

Ideally we'd have some international treaties that allow the top labs in America and China share information and coordinate as systems become increasingly capable.

Bizarre there are still denialists in places like this.

(It'll probably be masses of demigods, not one Singleton God, but obviously it's going to be some absurd frightening situation akin to this within our lifetime assuming no global AI ban.)

No.

No centralized party can be trusted with such immense, unprecedented power. Despite that, I still consider both Anthropic and OpenAI some of the least untrustworthy possible forces for this. (I am one of the only people on Earth with this opinion, but whatever.)

Decent odds things will go bad under their stewardship, but even higher odds with lots of other entities. Dario and Sam at least fully grasp the enormity of what they're about to be responsible for. Zuck does not. Xi Jinping and Donald Trump may or may not but in any case would use that to effect great harm.

Wanna make a bet with me on where the models will be at by 2030 when it comes to software development as well as most other white collar labor

The "think they're building the Machine God" part and "think they're the only ones that can be trusted to build it" parts are true.

Controversially: I agree with them. They're correct to think that. They are, and they are.

At the very least, the former is true even if the latter is murky. I trust them over Facebook and Google and China and open source "raze the commons" labs.

I trust Anthropic more than OpenAI when it comes to ASI catastrophe prevention, but I still trust OAI more than those other parties. But both still might fuck it up.

I just very strongly suspect it. There will be a plateau but a few years after, that, too, will be surpassed.

We are extremely early in the history of AI and even the history of LLMs. I'd easily make a bet with anyone on that.

Here's basically the most anti-Russia account on Twitter agreeing with you and me: https://x.com/P_Kallioniemi/status/2109542508682248478

Seems to likely just be a standard socially constructed folk belief

I'm not denying that Russia spreads disinformation and propaganda. They definitely do. I'm just skeptical knocking out these data centers is necessarily creating the effect people are anecdotally claiming to observe.

No, I know. That doesn't necessarily mean these Yandex data centers being hit affected that yesterday and today, though. For a variety of different possible reasons.

All of these arguments can easily be restated to apply to pre-AI coding, too.

Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.

I hate Russia with a passion and strongly support Ukraine, but this kind of convenient narrative always makes me suspicious. (I use Twitter daily and haven't necessarily noticed anything.)

Reminds me of when 4channers said 4chan's daily interracial posting went way down on October 7, 2023. (It didn't.)

I could believe this, but not before I see a good empirical analysis. Until then, it just feels like typical folk gossip.

Overall correct, but you're kind of missing some of the cat-and-mouse details for fingerprinting centered around infosec and anti-fraud purposes rather than generic ad-related tracking. That stuff can get way more intense, and is constantly changing based on new techniques developed, countermeasures from cybercriminals, and mitigations from browser developers.

I am not claiming to be a great forecaster, or anything, but to me "it sucks now, it's gonna get better, and then it's gonna get even better" was already apparent to me as highly likely starting around GPT-3. (The rationalist bloggers apparently already knew it as of GPT-2. Scott Alexander and Gwern, for example.)

I did not know when it would become so practical that I'd not feel much desire to hand-write code, but I was pretty confident it was coming.

I question the epistemology of people who did not realize this by around late-2024. (Sonnet 3.5 was out and could do some things okay.)

It's defensible to say "October 2026 is not yet the time when we should be 100% deferring code writing to AI (let alone not even looking at the code any longer)". But the writing is very obviously on the wall. It's likely not going to take longer than 10 years.

Also, not to sound like a naive hypemonger, but: in a decade I'd bet a ton of money the best AI systems will make strange mistakes of this nature at a far, far lower rate than they do today. They will gain a more holistic and more human-like perspective about each task.

(even if it's through some silly means like explicitly talking to themselves like "if I were a human doing this, what [... 5 million tokens in 2 seconds ...]" but also of course if they crack ASI and get something more efficient and intelligent than a human brain by then)