roca

The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.

With stolen cloud accounts and credit cards you can rent GPU and TPU time from the cloud providers. Stealing enough to train a new model might be hard but stealing enough to do inference with a frontier model would not be hard.

It's not that simple; the goods of AlphaFold are mostly not yet realized, and some negative effects of AI are being realized.

More importantly, as I tried to make clear in my post, the main reason I quit is not specific fears of AI doom (I'm uncertain about that), but my belief that AI is progressing too fast.

Google (unlike Meta say) makes a lot of money from a lot of things that have nothing to do with the addictiveness optimization.

I don't believe at all in the idea that cheaper LLMs will somehow "democratize LLMs". Rich people will always be able to buy far more compute than poor people, and we know how to turn more compute into more capability (bigger models, more reasoning and more agents).

The only way to address wealth and power inequality is (in democracies) to vote for governments that will fix it.

You have a fair point. I am financially secure so it's not a great sacrifice.

It was psychologically difficult to give up the opportunity to make more money at a fun job, though.

Thanks for the thoughtful comment. I go on this sort of trip pretty regularly so I don't expect this one to reorient me dramatically. They are good for my mental health though, for sure.

I think there are pretty clear mechanisms by which AI (a few model generations from now) could bring down civilization as we know it. One of them is a runaway cybersecurity scenario where hacking one set of systems quickly enables the AI to hack more systems (via supply chains, stolen credentials etc) and also host more instances of itself, and eventually everything Internet-connected that's not invulnerable just falls apart, which probably includes most electrical grids and a lot of other infrastructure. Yeah, the wilderness and countryside would still be there, but I for one can't live there without returning to the supermarket regularly for supplies. A lot of people would suddenly have to live (or not) without their meds. It would be bad.

I didn't, and still don't, think Google was unambiguously good or evil. They did dirty stuff with Chrome, they do addiction optimization, but they also provide great free services, fight endless wars against spammers, and provide reliable secure infrastructure to a lot of other people, and lots of other good things. I would never have joined the Chrome team, or worked on addictiveness, but I felt fine about working on optimization projects to make Google more efficient, which is basically what I did for my first few years there.

It's pretty hard to live in the world and be completely disconnected from anything unethical. Jesus doesn't seem to demand that (e.g. he didn't tell centurions to leave the Roman army).

> Are you aware tech bros have been saying this same thing about every AI advancement since the 1950s?

Whatever AI people have been saying for the last 70 years, AI capabilities have improved stupendously in the last 10 years and there is no sign of the rate of improvement slowing down. I would like to deny it, but I can't.

FWIW I was in the GDM GenAI unit. Not exactly working on AI directly, or I would have quit earlier, but in the middle of it.

Yeah, it's an age-old question. My high-level answer is, God's plan will be done regardless of what I do, but my actions determine (at least from my point of view) whether I'm aligned with the plan or not. And I want to be aligned with the plan.

(OP here) FWIW I despise the addictiveness, er engagement optimization that Google and others do. I have said so inside and outside the company many, many times. I would never have agreed to work on that stuff myself. One of the hobby projects I would like to work on is exploring how to use personal AI to beat those algorithms.

I respect the argument for not working for a company that does addictiveness work anywhere, even if you don't work on it yourself. But that's not where I drew the line.

Also FWIW I disagree the danger of AI is abstract. A lot is happening right now, in education for example.

I could have explained that better, but it was already getting too long.

Personally if I could Thanos-snap AI out of existence I would, but as long as it exists, it seems better on balance to make good use of it where possible than just try to ignore it.

"Focusing more on AI stuff" ... not really? Of course I was using AI a lot in my work at Google, as well as contributing to AI acceleration. Now I won't be contributing to acceleration and I can be as cautious as I want to be about using it.

Indeed, that message was for my coworkers. HR gets a form.

That would be ironic! But there's no chance I'll work on anything that accelerates AI. If I was OK with that, I would not have left Google.

I do expect to use AI in my work, as I indicated in my blog post. I haven't (yet?) reached the point where I consider that anathema...

Too bad the RAM crisis makes GC languages like Go far less attractive.

It's not that.

Especially with combined with reverse-execution in rr or UndoDB!

Record-and-replay debuggers like rr and UndoDB are designed for exactly this scenario. In fact it's way better than logging; with logging, in practice, you usually don't have the logs you need the first time, so you have to iterate "add logs, rerun 600 times" several times. With rr and UndoDB you just have to reproduce once and then you'll be able to figure it out.

We have explored that sort of debugging/visualization tool in https://pernos.co. We built it before the age of genAI, but I think for coming up with powerful visualizations AI is neither necessary nor (yet) sufficient.

You are correct. I worked on this for years at Mozilla. See https://robert.ocallahan.org/2007/02/units-patch-landed_07.h... and https://robert.ocallahan.org/2014/11/relax-scaling-user-inte... for example. Some of the problems were pretty hard but the Web ended up in a pretty good place --- Web developers pretty much don't think about whether scaling factors are fractional or not, and things just work... well enough that some people don't even know the Web is "free-scaling UI"!

It's depressing how many people are enthusiastic about making humans obsolete.

[rr developer here]

Undo has cool features like Live Recording that we don't have in rr. They don't need access to the hardware PMU which is a big advantage in some situations. They can handle accesses to shared memory in cases where rr can't. https://undo.io/resources/undo-vs-rr/ is a good resource.

Huh. I would pull out my phone, hold down the power button, and talk to Gemini Live. That's shipping today.

Altman apparently doesn't know what he's competing against. Not a good sign.

The least plausible part of this is the idea that the Trump administration might tax American AI companies to provide UBI to the whole world.

But in an AGI world natural resources become even more important, so countries with those still have a chance.

I don't know where that ranking comes from. It also matters that attackers adapt: UAF exploitation is harder than out of bounds, but it is well understood, and attackers can switch to it, so shutting off one source of UB isn't as effective in practice as you might expect.

And COBOL.

Writing new graphics drivers in Rust will definitely be helpful, and is starting to happen.

The safety of LLVM and GCC need not be a priority... they're not normally exposed to untrusted input. Also, it's a particularly hard area because the safety of generated code matters just as much as the safety of the compiler itself. However Cranelift is an interesting option.

No silver bullet here unfortunately... but writing new infrastructure in C or C++ should mostly be illegal.

There isn't anything new here to defend against lifetime-related UB. For that it simply references https://arxiv.org/pdf/2503.21145, which is just a summary of existing dynamic mitigations --- which don't fix UB at the language level, impose performance tradeoffs, and in the case of pointer integrity, require hardware support that excludes e.g. x86.

Look at it this way --- mature products like Chrome are already doing all of that wherever they can. If it was enough, they wouldn't worry about C++ UB anymore. But they do.