Comment by haswell | Hacker News Reader

Comment by haswell | Hacker News Reader

Parent Full thread

The A in AGI stands for Artificial, and the result is by definition not human. Thinking about this problem in terms of human dynamics is a non-starter, and employee/manager relationships as we understand them are irrelevant or misleading at best.

> Do you think you'll be able to prompt AGI "do business things" and it'll just carry on in its virtual world that will somehow translate to real value?

Simply: yes. And much more.

This is the general concern about AGI in a nutshell.

If an alien ship shows up tomorrow, and its occupants express a desire to coexist/cooperate, and it becomes clear that they are significantly smarter than we are, what concerns come to mind?

What if suddenly every company in existence is angling to benefit from this higher intelligence, and wants to proceed with full inter-species integration/collaboration full speed ahead, without fully understanding the goals of this other species?

What if we later learned that this species prefers extremely hot climates and thrives on high CO2/low oxygen environments? Would we be so willing to participate in a short term business optimization initiative?

What if we also learned that this species is remarkably similar to humans in that it will compete for various forms and degrees of power while ignoring the long term ramifications?

Given what we know about our own species, the prospect that a super-intelligence might be built by and based on it should be terrifying.

Ai isn't an alien. It doesn't have autonomy, will or desires. It's a set of numbers arranged in a particular way such that someone can use it to produce different numbers that can be interpreted as meaningful. That's all it is.

How do you get from that to aliens controlling our climate?

Give me one reasonable path. Will the numbers start changing themselves and pop out of the computer and float around and start moving things?

This AGI doomsday nonsense is becoming more of a religion every day.

> Give me one reasonable path.

Actual AGI appears, and it has the ability to improve its source code. It becomes more intelligent than humanity and is misaligned to human goals.

These goals will the mathematical representation of what the AI is trying to optimize, and there are many ways for this optimization to cause negative effects.

We haven't even reached AGI, yet the LLM GPT4 was already able to act in the real world by manipulating a Task Rabbit Worker.

> Ai isn't an alien.

The goal of the alien analogy is to provide a thought experiment that allows us to expand how we think about AGI without getting stuck in preconceptions.

It is not literally alien in the sense that most people think about when contemplating extraterrestrial life, but there are some strong parallels that are informative to consider.

> It doesn't have autonomy, will or desires

Current-generation AI tech is already teaching itself how to walk [0]. This is not a robot that follows deterministic rules pre-defined by a human, or a robot that was trained to walk. The robot trained itself.

The AGI conversation is a conversation about the future where each of the specialized fields of AI converge. Much like we are massively complex interdependent systems, I suspect AGI will be the culmination of integrating focused fields of AI technology into a single system.

Will and desire are human feelings that ultimately stem from our goal-oriented behavior. The absence of these feelings in a robot do not matter if the robot behaves in a goal-oriented way. Problems emerge when its goals are not aligned with our goals. This isn't about some hokey robot overlord scenario, but one where a robot sees humans much like humans see ants when going about our days.

Most humans don't go out of their way to destroy ant colonies, but neither are we very concerned when we run the lawn mower over one. Malice is not required for a misalignment of goals to be a serious issue.

> This AGI doomsday nonsense is becoming more of a religion every day.

None of what I've described is inevitable, but all of it is possible. We're now in the moment where we'll decide what is next, and if we'll take this seriously enough to bake in appropriate safeguards. I think Stuart Russell has important things to say about this [1].

While I'm not a doomer, I'd strongly encourage you to dig deeper into this subject. I was shrugging most of this off for awhile, but things have changed.

- [0] https://vcresearch.berkeley.edu/news/step-step-berkeley-robo...

- [1] https://www.youtube.com/watch?v=ISkAkiAkK7A

The flaw here is that the aliens are significantly smarter. This logic doesn’t translate in your comparison.

The excel “aliens” are significantly smarter than humans when it comes to basic organization and math, but they still require humans to generate value.

Similarly, AGI is much smarter for a set of tasks, but it still requires humans to generate value. LLMs are astounding/fascinating in what they are able to do, but they don’t replace employees end-to-end (except for a few niche industries).

Better tools will be made with AGI, and the outcomes will better society. For a long time now, humans have required computers/AI to compete in business, and this will just be extended by the new AGI. Humans+AI will always be better than either alone.

> Humans+AI will always be better than either alone.

What makes you confident in this belief?

The confidence was for rhetorical purposes really. There’s no empirical basis to make such a statement.

I do, however, believe it. The current state of AI is so far behind the human mind that I think it’s reasonable to say it will never surpass the Human+computer combo.

It’s like talking about humans traveling outside of our galaxy. I guess, theoretically, it’s possible or technically even likely with a large enough amount of time. Theorizing that far into the future is meaningless though, as far as I’m concerned.

> Similarly, AGI is much smarter for a set of tasks, but it still requires humans to generate value. LLMs are astounding/fascinating in what they are able to do, but they don’t replace employees end-to-end (except for a few niche industries).

What you're describing sounds like an early iteration of AGI. But unless we stop the research, progress will continue and the limited AGI you describe is not where things will stop.

LLMs are not AGI, and their limits are unsurprising as such. But the limits of LLMs are not informative about the broader concerns of an eventual AGI.

In each of your descriptions you've limited the scope of what an AGI can accomplish, but those limitations need to be justified. If we continue to iterate, and especially if there's ever a moment where the computers surpass our own understanding, there is no reason to believe that those limits you imply are absolute.

This is totally valid and I misinterpreted your earlier statement to say that we’ve already arrived at what you’re talking about.

It’s a tale as old as the semiconductor really. Ray Kurzweil’s “The Singularity is Near” only with a different spin that incorporates the latest tech.

I do, however, personally believe that computers won’t ever surpass the Human+computer combo. That is to say that computers will never be so dominant that our involvement is useless.

Pontificating about the future wasn’t the point of what I was saying though, rather that the current state of AI (even with the incoming few years of generative AI applications) is a far stride behind the current Human+computer combination.

Reply on news.ycombinator.com