Hacker News Reader

No one's at the wheel. It's fair (to a degree) for companies to push for short term gain, but the legislative counterbalance that is supposed to care for long term effects no longer exists.

It's not only AI. Betting is on the rise, adults and minors are being hooked by dopamine loops, social networks are causing well measured negative effects, children's educational scores are sinking worldwide, appliance manufacturers engage in surveillance...

It truly feels like the wild west.

> but the legislative counterbalance that is supposed to care for long term effects no longer exists.

In the US. When the inevitable legislations start popping up worldwide, don't be surprised when somehow the American consensus will be to do nothing.

You have to remember that an American consensus in it's current condition is really just a consensus among the corporate interests that have captured the government.

In the US, yes, and the US is still in the position to boss everyone around.

Two other stories in this vein: "The Machine Stops" (1909) by E.M. Forster and "Pump Six" (2008) by Paolo Bacigalupi. Very different milieus, but both deal with decadent, hedonistic futures where humanity coasts on legacy systems engineered by long-dead experts until they begin to break down and no one alive knows how to fix them.

Thank you, I'll try and find those.

> Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work.

fwiw, i think literally all largish corporations have been like that pre AI too? i can gurantee you that nobody understands how the whole system works at all of my last workplaces either...

people just know their small part and hope for the best - so from a devils advocate pov, its already been proved that its sustasinable. now wherever its _worth it_ is another question. large employers are - on average - incredibly wasteful with their resources. not to anthropics degree, but still.

if youre boostrapping, and VC token subventions go away... i suspect it'd be not worth it to develop like that. But i guess we will find out in a few month to 1-2 years

I think the difference is whether agent availability in the future is guaranteed. I agree that no one had the whole picture before, but at least theoretically you could replace x employees with another workforce who in time could be expected to match the understanding and output of the original workforce. With agents, if capability is consolidated with a single organisation and that organisation changes it's terms of business, you cannot replace the army of AGI-agents with a human workforce. It's a single and total point of failure.

Your hypothetical assumes that AI will suddenly vanish in its entirety, and we are left picking up the pieces.

Yet even without AI, humanity's most complex creations are not understandable by mere single humans, or even groups of them.

For example, the US Healthcare Insurance system runs on the collective power of humans (and likely an increasing bit of AI), but no one human can understand the system as a whole.

what if it just becomes prohibitively expensive?

Humans having understanding of the system went out the window hundreds of years ago at the latest. The "Renaissance man" era is long gone. I know you said "collective understanding", but that's not a real thing. Humans aren't borg. Our entire lives have been spent in systems we can only understand in piecemeal if at all.

I think in many areas we are going to see a renaissance of the "Renaissance man" since AI makes it possible to do so much more as a single person.

The Renaissance Cyborg.

I wouldn't put it exactly like this, but I agree that many systems chugging along today (such as the COBOL running on IBM metal that powers banks) are not something people try to (or can) comprehend all at once.

Is that really a problem though? We can solve problems empirically even if we don't keep the entire system's workings in our heads.

You can solve problems empirically, but you can never guarantee you've hit all the edge cases the original code did. That's fine for non-mission critical systems, but would you be ok with your passenger jet onboard systems having been rebuilt from empirical observations?

That's a funny example, given that the Wright brothers were overwhelmingly empirical compared to their competitors and succeeded as a result, because the foundation of aerodynamics was fundamentally flawed at the time. In a real sense, those systems were built from empirical observations, and most human systematizing happens post-hoc.

I agree that formalizing observations into a system can lead to additional insights, though it would probably be hubris to assume the formalization is faultless.

Speak for yourself. I have a pretty good understanding how the system works from the transistors up.

Do you good understanding of where your food comes from?

How about the dollar you earned, spent, invested? Do you know how it works?

How about the clothes you wear?

This is just a tiny subset of the enormous complexity we're immersed in, and no individual or collective has a complete understanding of it. If you do, you're truly an outlier and GP's point still stands.

Some individual has an understanding of each of those areas. Why does one individual need to know the entire food chain? The discussion point is about each specific area.

Even for the COBOL based systems someone understands it well enough at the conceptual and business level. And when there's no one to the company migrates and modernizes.

You are conflating the subject matter expert who understands one area well and others not so well with a general newbie who understands nothing well

> The discussion point is about each specific area.

I don't think it is. @squidmaster specifically noted, 'The "Renaissance man" era is long gone.'

Yes for all the examples you gave. None of those should be beyond the understanding of a curious person living today.

What happens to that understanding when you're gone?

The agents only have to be better than we are and we’re not very good. Same argument as self-driving.

Don’t fear change.

Unless something dramatic changes and it'll become cheap to "constantly re-educate" the models to learn the new fundamental facts about the system (ideally with a small lag of maybe a few days like with humans) I see no reason to think that LLMs can have a profound understanding of how business systems work of a remotely comparable quality to humans. Just adding the context about the system already reduces it's capacity significantly according to what I can tell. Not having large context helps somewhat at a cost of having to explain lots of task-relevant things to LLM over and over again...

> I see no reason to think that LLMs can have a profound understanding of how business systems work of a remotely comparable quality to humans

My manager was dragging me through useless projects that went nowhere despite multiple people telling him "it ain't gonna work" while shooting down proposals that later turned out to be "high priority tasks required directly by upper management". And he's a well-paid, educated employee with years of experience. Don't even get me started on delivery guys who cannot read a simple instruction like "do not ring the bell, leave the package on the doorstep". They just follow a 50-line script and that's all they can do.

Whenever you mention that AI won't reach human intelligence, remember that this is the benchmark. Not Alexander the Great, it's the millions of people throwing their life savings ($1000 each) into Labubus. Long time ago there was a series of articles in my country about the costs of wedding receptions skyrocketing and some people taking out loans to pay that, and someone made a meme "robots will never have human intelligence!" with reply "bro, you took out a loan to pay for a wedding, I don't want your human intelligence".

Denying this reality is pure cope.

Can I poke at this quick?

You are showing disagreements with your priorities (mostly money) as unintelligent. You're showing a lack of care and conscientiousness as unintelligent.

You're reducing intelligence to alignment with money and following instructions. Under that framing the argument is already won in favor of AI amd some vague effective maximizing of a cost function.

Many jobs are like this, so it's fine for employment. Maybe the vast majority of mankind was never suitable for reliable employment of the "cost function" style.

But I know two things:

1. Centralization should be avoided. The reality that AI can do most jobs is not a reason to concentrate power behind those portraying that or building AI. We still want local, ideally individual, empowerment. I don't see enough said about this. Maybe we can even say the system must be one people wish to opt in to, to maximize profit.

2. Intelligence is being very narrowly defined and humans value more than an INT score in the RPG sense. The more I meet people outside my belief circles, the more I realize the intelligence of some actions, in the sense of "logically deduced from a higher principle" not "aligned with my western white collar principles". The ultimate role of culture and humanity IS this principle exploration and to say we've reduced that exploration to "maximize X" is crazy.

At some point deciding fate of the company or system or state flows from principles embedded in cultures and that's in danger of being flattened away by optimizing outcomes with machines.

Vague and wishy washy, but its the clearest I can state it.

> At some point deciding fate of the company or system or state flows from principles embedded in cultures and that's in danger of being flattened away by optimizing outcomes with machines.

If the principle embedded in the culture of my manager is to do dumb shit that goes nowhere, then I'm not sure what goals we're achieving.

I think we're at the "what's water" stage of the discussion

That was a speedrun lol

I think this could be summarised as alignment being conflated with intelligence - i.e. what seems unintelligent to some is just misalignment with their own values.

Well I'm obviously not comparing to average human intelligence (and negligence), but that's actually a good reminder about one other important aspect of how living organisms are very different from machines -- humans have very high range of intelligence, strength, etc, and the best humans are usually much better than the average in all areas of life. Computers and software are much less diverse.

You’re a simpleton who lacks empathy.

Denying that reality is pure cope.

Online / continual learning will solve this problem. Test time training has already proven to be effective.

> The agents only have to be better than we are and we’re not very good. Same argument as self-driving.

Self-driving requires less holistic thinking than imagining, building, and evolving large scale systems. It requires abilities that LLMs do not have. It requires non-symbolic reasoning, intuition, etc. that are beyond LLMs.

Current self-driving systems also require periodic manual intervention from remote operators https://www.reuters.com/technology/waymo-defends-use-remote-... , for that matter.

Intuition is just system 1 thinking. Nothing that doesn't already exist.

> Intuition is just system 1 thinking. Nothing that doesn't already exist.

Intuition exists outside the human brain?

I'm surprised this is even a question and the fact that you used human and not even mammal to phrase your question. Do you think intuition is a unique human trait?

> Do you think intuition is a unique human trait?

No.

Then you must think it exists outside the human brain, right?

But an appropriate level of caution about change is necessary. Change isn't always good https://www.mentalfloss.com/science/environment/clair-patter... , and it doesn't always deliver what it promises or arrive on schedule.

"Don't fear change" is a religious statement, no different than "never accept change".

Imagine I am 200IQ, and I can solve things you can't comprehend.

You appear to be saying I should this be able to run any system, including the entire United States fed govt, and anyone who even questions if that may have bad effects should "not fear change".

The industrial revolution was Change, but so is cancer.

> The agents only have to be better than we are and we’re not very good. Same argument as self-driving.

> Don’t fear change.

Change is inevitable.

What I understood from the comment is that Op fears the boiling frog scenario. [0]

0 https://en.wikipedia.org/wiki/Boiling_frog

Imagine a country where no one can drive, and all transportation relies on a few AI monopolies.

If you don’t pay them, you can walk by feet. If they ban you, you can walk by feet. If their infrastructure is damaged, no one and nothing will be able to move.

We have that now in several systems, maybe most, it just isn’t AI. If your water has lead in it, you’ve got one highly incompetent local government to remediate it. If your health care goes out of network with the only hospital system in your small city or town and you have a heart attack you’re fucked. If one security software pushes a bug to production the airlines shut down. Etc.

Single points of failure are a fact of life.

AI offers relative democratization. What’s going on with model competition is giving us more choice, not less.

I'm reminded of the visual novel "Root Double -Before Crime * After Days-" where one guy in town having a manual petrol-powered car instead of using robotaxis is a significant plot point.

Battlestar Galactica also relies on a pre-computerised starship as being the only hope for humanity.

And it had a happy ending!

but all of their civilization got destroyed (in the beginning of the series in case you haven't watched it).

Get your own compute, require and force openweights and right to compute. Make sure NV and AMD keep making prosumer hardware that can run Kimi k3

You’re more than welcome to keep writing code by hand. It’s not illegal… wouldn’t recommend it though.

Good point about infra, but that’s already the case with Cloudflare, GitHub, aws, etc

You're more than welcome to wake up in a country that is 100% dependent (in all spheres of life and economy!) on a few monopolies. I'm not sure that thing will be a country anymore.

BTW, when (not if, but when) a war happens, they will become very nice targets to render this exact country completely disabled.

Your last point is the one that I find most compelling.

Humans are autonomous, distributed, and adaptable. In a package that weighs ~150 lbs, with very low energy requirements.

You literally cannot wipe them out without a sustained and incredibly expensive ground campaign, and even then it's not likely.

AI in a data center on the other hand... So easily disabled.

I can easily imagine, say, Russia destroying a few datacenters not with nuclear missiles, but with some relatively trivial local ground-based FPV attack or whatever.

And voila. No cargo moved, no code changes or fixes, no medical diagnosis made,etc. A whole country is disabled.

There are already tons of countries in Europe and Asia that are dependent on a few monopolies. People still live their lives just fine.

Re: war.. that’s why US spends orders of magnitude more on military than anyone else. If any infra is hit on US soil then im heading to the bunker because most of the world will have already been reduced to dust.

Yes and the whole sovereignty discussion started exactly because we realised the issues that are now surfacing with that.

It's nowhere near "destroy a few buildings and EVERYTHING will be disabled" at the moment.

yeah, you just have to flip the wrong switch at us-east-1

There was a book I read maybe 10-15 years ago about a world where governments Essentially dissolved and in place there was nothing but mega corp monopoloys - Jennifer Government was the name ( if you ever played the MUD Nation States yk)

I thought that was an interesting idea, but not something that could ever happen..

Or buy a cheap Ebike.

> Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work

That's the point, they are setting up the economy to be run by agents and they trust that it is possible. Step 1. Teach AI human knowledge (2022-2023). Step 2. Make AI agentic (2025). Step 3. Start autonomous research and create own corpus of knowledge (2026).

Step 3 will keep compounding (as we are witnessing) and most people will be left behind. I'm not against a revolution, I'm all for it.

AI is far from having all human knowledge - long way to go to run everything in the economy by (robotic) agents.

Pre-training already makes up a very small cost of the training now. Post-training makes up ~90% of the training cost as modern LLMs are trained using RLVR on synthetic data. People still think that a lot of it is coming from scraping wikipedia like it is 2023.

Not sure how that gets specialized off-line knowledge in certain manual tasks, for example.