achierius

This was already a thing before AI, the term is "variant analysis". But it's been way juiced up by LLMs and is 100% a major cause for concern.

> the patch itself gets only cursory (claudy, codexy) review

FWIW Claude is actually quite good at discovering variants of existing bugs.

The simple fact is that you can't just open an app and then let it destroy the lives of people with absolutely no connection to it. The company itself must be responsible. Saying "the optimal amount of fraud is non-zero" is only acceptable if the cost of that "non-zero" amount is borne by the company itself; otherwise the result is plainly misaligned incentives.

Extant lawyers are unlikely to be replaced in the immediate future, if only thanks to regulatory barriers.

What is that the bottleneck for, though? Certainly, it's tautologically the bottleneck for "humans understanding math"; but what about things like using math to drive discoveries in other fields, or for practical work in CS, or applications elsewhere? I fear that practically those aren't bottlenecked on human understanding.

> move somewhere cheaper to live

Unfortunately this often means moving away from family. What are you going to say to the kid whose parents toiled to get him through college on the expectation that he would be able to take care of them? "Sorry Mom, I have to move to North Dakota to work on the oilfields"?

A few notable exceptions?

Are phones an "automation of existing jobs"? Maybe couriers (??), but most of what people use them for goes far beyond that.

What about the software used to drive trains or autopilot planes? We still have conductors and pilots, but now the outcomes are safer on average and more efficient.

Or something like Google Maps -- all that replaced was the paper booklet people used to keep in their cars, it's not like we used to employ 'navigators' to go out on a road trip.

Or something like Intuit QuickBooks: there's still an accountant in the loop, and if anything more accountants today than there were a few decades ago.

Certainly some software is essentially automatory in nature. But to characterize 'nearly all' as being such just betrays a lack of awareness for the rest of the field.

> That one is somehow deserving of sympathy and that of mathematicians is not

He didn't say that mathematicians aren't deserving, he was griping that people said this about SWEs. Do you think mathematicians are deserving but SWEs aren't?

Anthropic and OpenAI are American companies. Why would you be so surprised that discussion on them considers the plight of Americans? Do you think poor Americans don't deserve healthcare?

> the problems of your healthcare insurance system are not a worldwide phenomenon

That's great for Europe! But unfortunately people are still dying here. Why are you upset that people are complaining about that? Do you not think they should be able to discuss their problems?

It would be. But that's not going to happen.

There's very little diffusion of responsibility here; almost everyone at the labs is directly working on building incredibly dangerous systems that they themselves admit to being unable to control, and they almost all have the financial means to stop working if they so chose.

Why do you trust them to build it? They've already demonstrated that they're unable to prevent breakouts when training today's level of model, why will they suddenly be more careful / "smarter" when training tomorrow's?

You say you trust them more than Google, but Google's actually kept their models under much better lock-and-key. But even if we do take for granted that they're more trustworthy, that still doesn't make them actually trustworthy.

I'm sure that I'm not!

But it's good that they say this out loud; the sooner that the public comes to realize what monsters are in charge of these labs, the better.

When labs bring up the topic of "warning shots" I always think of this: https://hollyelmore.substack.com/p/hollys-basilisk

Why do Altman and Amodei seem to think that they'll escape punishment if a "warning shot" that kills Americans were actually to come to pass? Certainly in a just society they'd be on trial already.

You can engage with society without being 'well adjusted' to it. The Franciscan Friars are a great example; St. Francis was obviously not interested in 'conforming' to what he saw as a society in need of reform, but he also didn't try to hole away in the hills.

> Things are currently moving fast. They cannot move fast forever.

This is a supposition that I fear will soon be proven false.

Being the 'logical culmination' of the Industrial Revolution does not mean you've been automated (and thus suffered the alienating consequences), rather the opposite: you're currently on the un-automated cutting edge. Your intangibles are exactly what others have lost, and you personally will lose, with further progress in automation.

You're right, and that's a bad thing. AI is nothing fundamentally new, but its extremity is making many people aware of the truth that's been there all along. There's no contradiction in that.

> Do we see strong moats that are going to cause monopoly in AI?

Ownership of the capital assets used to train and inference new models. Yes, we may end up with more than one firm. But as we see with big tech today, a small number of fantastically wealthy firms in "competition" does not an open market make.

Objectivity? Why would you want favorable reporting for the machines they're building to replace you, and, by their own admission, potentially kill you?

The only bias here is that we're still covering these things like business ventures and not criminals.

I'm sorry, do you understand how the medical industry works? It wouldn't be one cure, it's going to be dozens of treatments, each of which will be incredibly expensive.

Or else how do you explain Danyelza? Used to treat neuroblastoma, costs upwards of $1m per year. Do you have any proof that this will be different?

You're calling realistic people conspiracy theorists. Whose side are you on?

What makes you think we'll be able to afford them? You won't be making any money anymore, hope you've saved up!

As a compiler engineer, it's not hard to find opportunities for the compiler to produce better code. What's difficult is turning those into generalizable patterns without introducing bugs or regressing performance elsewhere. You're right that as time goes on hand-written assembly won't change, but I see that as increasingly less of an issue now that we have LLMs to pour over and re-analyze what's going on as new releases come out.

Ethiopia has been a great power since then too: under Constantine I (c. 1450) it was the preeminent power in East Africa, and while the kingdom subsequently fragmented it managed to recombine by the 1800s in time to take a stand against European colonialism. None of that was because of a "Greek minority", African people are perfectly capable of governing themselves (with the right social structures in place, as with all people).

Silly emotions? I'm not the one gleefully celebrating the ruin of others (including families). You've chosen to 'other' a broad group of people with diverse experiences and backgrounds. "Those tech people" certainly include some people who called manufacturers 'dumb', but it also includes many (most?) who didn't, and even some who used to be "manufacturing people".

You say "who cares if they suffer? They're obviously all just bad people". The people who attacked blue collar workers said "who cares if they suffer? They're obviously all just dumb people (and thus bad)." So the bulk of people who you decry are not guilty of what you claim. But you are!

WW2 engineered things we've never imagined. Are you looking forward to a redux?

Papal excommunications present facts and data. These things aren't exclusive.

Can you not see how you're doing the same thing now? Not all people in tech so dismissed the struggles of others -- but you are.

Live out of a van with children? Is that what you're suggesting people do? That's monstrous. You would rather put the weight of the world on a poor family's head than one iota of responsibility on the ruling class.

The Parthians sure did :)

That's a very broad statement; the reality depends on the specific order and monastery in question. To be sure monasteries are wary about all who inquire, thus the usual two-year neophyte period. Beyond that some will expect more from novices, other less, but there are many stories of broken people retreating to monastic life even today. Indeed many (within the Church) expound that as a key role of monasteries, providing a place for people to escape from the world. If you just want to live a Godly life, you can do that in society; monasteries are precisely the place you go to do so outside of it.

Actually that's not generally true. Janitors at least work 12 months a year, many schoolteachers are paid ~9 despite being expected to work through them (on lesson prep &c).

Not everyone's had the luxury of saving for years and years. Some people just got into the industry (remember 'learn to code'? not to mention recent graduates), others had to work their way up through lower-wage roles (e.g. contracting) just to get to a place from where they could hope to save. Other people have family to take care of and can't just throw everything into their savings.