LLM Powered Autonomous Agents
lilianweng.github.io
lilianweng.github.io
So if you ask ChatGPT "Describe Berlin", what happens is that the NN is called 6 times with these inputs:
Input: Describe Berlin.
Outpu: Berlin
Input: Describe Berlin. Berlin
Outpu: is
Input: Describe Berlin. Berlin is
Outpu: a
Input: Describe Berlin. Berlin is a
Outpu: nice
Input: Describe Berlin. Berlin is a nice
Outpu: city
Input: Describe Berlin. Berlin is a nice city
Outpu: .
Berlin is a nice city.
And how does the NN represent the token at the output layer? Is it a binary representation of the token number?
Or does it have a neuron for each token it knows and ChatGPT takes the most activated neuron as the answer?
But I personally think it's a coincidence, and it just so happens that 50k tokens are enough for the level of complexity the models have right now.
Tokens are part of words, approx 4 characters or 75% of word.
It gives a list of tokens with their probabilities on output.
It's a short list with highest probabilities.
Temperature controls which tokens to pick - usually 0% = top one only (consistent results), closer to 100% means more randomness (more "creativity").
It's "glorified markov chain" in the same sense that sqlite is just "glorified bubble sort".
For example: if you limited all inout/output to the same 100 words, could you stay within the token limit permanently?
The really exciting stuff is how they're doing that! Research into that question is generally called interpretability, and it's probably what I'm most interested in at the moment.
Holy.shi..
As I have been observing the ways AI struggles and the ways it produces output, I’ve had this growing suspicion that LLMs have been somehow showing us things about the human brain, but it wasn’t until this moment that I had any validation of the theory.
I’m very curious about the insights that will give us in to LLMs. My initial instinct is that it won’t work unless you have essentially the entire state remembered since each “point” in the LLM can lead anywhere. I’m imagining an experiment where the task is “Get a person who’s never been to this city before from point X to point Y without describing the destination and while providing as little information as possible”. Giving them the full map with directions is the equivalent to giving the LLM the full previous conversation.
Intuitively there is low-hanging fruit (just provide the directions of single optimized route), but it feels like it gets chaotic quite quickly after that. How many steps could you leave out of turn-by-turn directions before the person is hopelessly lost?
This is based on my current understanding of LLMs so YMMV, but I’m definitely curious about ways we can start from a remembered state.
At the moment, image models are mostly diffusion models while LLMs are mostly autoregressive. With a few other examples. Google has for example created both an autoregressive image model, Parti, and a diffusion model, Imagen.
It's still early for non-autoregressive LLMs.
* empty string
* Berlin
* Berlin city
* Berlin is a city
* Berlin is a city in Germany
* Berlin is the capital of Germany
* Berlin is the capital of Germany, located in the North East.
* Berlin is the capital. It is located in the North East of Germany.
* Berlin is the populous capital city of Germany. It is in the North East of Germany, by the river Spree.
* Berlin is the capital city of Germany with 3.6 million inhabitants. It is located in the North East of Germany. It's centred on the river Spree.
...
At every step, the diffusion slightly rephrases to add more information content and detail to that of the previous level, by inserting, replacing and deleting tokens.
It is usually more costly to sample from these diffusion style models (which is not the case for diffusion for images, as unlike text, images tend to have a fixed size). But, one might imagine this approach to scale up to generate texts of arbitrary lengths. Just keep on adding more and more detail on your text, and you'll end up with a book or a coherent novel.
Interesting. Can't you just start with 4K of randomly assigned tokens and tweak it until it's right (including the tokens for EOF), like they do for images where they start with noise and move towards an image?
But for this approach adding "noise" would be much less straightforward, wouldn't it? You'd have to have some way to work out how to take "Berlin is the capital city of Germany with ..." down to an empty string in a way that marginally removes detail. For some passages this could be rather difficult to do. I feel like gathering training data would be a huge trial here.
You could then greedily take the most probable single token, or select a bit more randomly (that's where temperature comes in) in a few different ways. You can do better still by exploring a few of the possible top options to select a high probability chain of tokens. That's because the most probable next token may not be the start of the most probable sequence of tokens. That's called beam search.
Other than that, things like gpt can't go back and edit what they've outputted, so it's much more like listening to someone talk from the top of their head and writing down everything they say. They can say something wrong, followed by correcting themselves. The fact that the output becomes the input makes it much more obvious why "lets think through step by step" helps them reason, they can do simpler things then see the simpler steps they've already done to answer the more complex parts.
There are other decoding approaches, other than beam search:
https://arxiv.org/abs/2202.00666
> Other than that, things like gpt can't go back and edit what they've outputted
Actually, they can, they may emit a backspace token [1]
[1] https://arxiv.org/abs/2306.05426, https://news.ycombinator.com/item?id=36425375
https://blog.fastforwardlabs.com/images/2019/05/Screen_Shot_...
The way most GPT models are set up, they don't edit, but there was a lot of work (made redundant by OpenAI) on using gap filling tasks (masking) to iteratively improve or construct or otherwise edit and formulate logic using the encoder-decoder models of yesteryear - in many ways, I think it would have been more exciting if that approach powered the likes of 'think step-by-step' results.
There's nice stuff you can do with the probabilities, a shame it's more restricted with the chat models. I hope they add some lightweight thing you could pass with your request to constrain the response.
However, it's important to note that the numbers which the models returns for the next tokens can only be interpreted as probabilities for foundation models. For fine-tuned models (instruction SL, RLHF) the numbers represent how good the model judges the next token to be. This also leads to a phenomenon called mode collapse.
> Human: Describe Berlin
> Bot: Berlin is a city in Germany. It has been the capital of Germany since 1871 and it's also known as "the city that never sleeps".
While working on smol-developer I also eventually landed on the importance of planning as mentioned in my Agents writeup https://www.latent.space/p/agents . I feel some hesitation with suggesting this because it suggests I'm not deep-learning-pilled, but I really wonder how far next-token-prediction can go with planning. When I think about planning I think about mapping out a possibility space, identifying trees of dependencies, assigning priorities, and then solving for some kind of weighted shortest path. That's an awful lot of work to expect of a next token predictor (goodness knows its scaled far beyond what anyone thought - is there any limit to next token prediction?).
If there were one focus area for GPT-5, my money would be on a better architecture capable of planning.
Having a tasking agent refining prompts seems a much robust approach.
Bonus if it can decompose the required data properly. I'm working on a world database prompt with limited success, but implementing the retrieval with prompts is time consuming and I've not much time. The idea is along the line of "translate the user question in SQL, you have a database of all known facts with a table for each entity" and then you handle the retrieval of each table data step by step.
I think then you'd be able to use the postgres planner as planner, but while I've seen project of postgres retrieving data from random sources I don't remember the specifics.
but that's two things: planning and execution of a plan. The plan is the dependency graph (!), assignment of priorities and shortest path is execution. This is very important in the context of agents, autonomous or not. If you want the agent to self-correct it has to understand that there can be multiple start points or multiple end points (or both) to backtrack and pivot. And as long as it is glorified "next token predictor" it cannot really do that.
Of course some tasks indeed are ifttt style linear-ish flows where next token prediction may prove to be adequate. However, if your agent is incapable of understanding non-linear flows, can it reasonably back off from one true way hen faced with such flow?
I at least do not want to predict the next token, I want to predict the next concept in a chain of reasoning, but it seems that currently we are stuck at using the same representation for autoregression we use for training.
Maybe we can come up with a better way to construct these chains once we understand the models better.
Edit: Typo
from u/gmt2027
>An extreme version of the same idea is the difference between understanding DNA vs the genome of every individual organism that has lived on earth. The species record encodes a ton of information about the laws of nature, the composition and history of our planet. You could deduce physical laws and constants from looking at this information, wars and natural disasters, economic performance, historical natural boundaries, the industrial revolution and a lot more.
and u/thomastjeffery
>That entropy is the secret sauce: the extra data that LLMs are sometimes able to model. We don't see it, because we read language, not text.
Language is a communicated abstraction of concepts, and it would seem that internal representations of those concepts can emerge from something optimized for token prediction. Or at least: internal representations of the speaker to be predicted, including the knowledge they may possess.
Imagine translating language into an optimized representation free from human constraints, doing autoregressive prediction in this domain and only than translate back.
As far as I understand current models, this is not yet how they work.
Deep Reinforcement Learning with an Unbounded Action Space - https://arxiv.org/abs/1511.04636v3
Efficient Planning in a Compact Latent Action Space - https://arxiv.org/abs/2208.10291
It's not everything involved in traditional planning, but it may be a framework to use more traditional planning algorithms on LLM output.
What you really do is more like: you've already formed a half-baked plan before the problem even fully registers, in one cognitive step of pulling the ready-made if somewhat fuzzy graph out of the depths of unconscious - and then you start doing all those things you mention, while trying hard not to be biased towards or anchored to that original plan-shaped blob of words, images and emotions.
I think creating that first plan ex nihilo and putting it into words is exactly what next-token-prediction can do, because that structure of options and dependencies and priorities and a path through them is something that's encoded wholesale in the latent space, and needs to be projected out of it. That's at least my understanding of what current transformer LLMs are good at - encoding every imaginable relationship between tokens and token sequences into proximity in a four- or five-digit dimensional space, and then projecting out of it on inference.
Going beyond that - beyond that first plan prototype, which may or may not be good, is getting increasingly explicit and thus increasingly hard. I think you can ultimately coax an LLM to do all the steps you listed and then some, by walking it through them like infinitely patient parent of a 3 year old - but not within the context window, so you need to split it into multiple invocations. This grows in cost superlinearly, as you'll have multiple results you need to summarize to feed into next stage, and if you're computing enough of those results then they won't fit into context window of the summarizer, so you need to add an intermediate level - and then eventually summaries may not fit in the context window of next step, so you need another intermediate level, etc. Context window is the limiting factor here.
And if you're to use LLM to emulate a mechanical/formalized process, then it's better to just extract structured information from the LLM, feed it to regular software optimized for purpose, and feed the results back. I.e. something similar to LLM+P approach - or similar to any time when you get LLM to "use tools". If it can call out to a calculator, it can just as much call out to some PERT tool to get your list of steps sorted and critical path highlighted.
On a tangent:
> identifying trees of dependencies
Are you sure those are trees? :). My pet peeve with all the hip software used to manage projects and track work in our industry, is that it treats work as decomposing into flat trees. In my experience, work decomposes into directed acyclic graphs (and in certain cases, it might be useful to allow cycles to make representation more compact). There's hardly a tool that can deal with it in work planning context, not unless you step over to "old-school" engineering (but those tools aren't cheap or well-known). Which is ironic, because at least when it comes to build systems, software devs recognize the DAG-like nature of dependencies.
However, transformers are not able to loop. In a fixed amount of computing steps, they need to predict. For long-term planning, we probably want something more recursive/iterative (but that will be harder to train).
Also today we can compensate that with wrappers (eg. Langchain), but ultimately the machine will learn end to end.
Another case that a friend of mine built - summarise a stream of news from Twitter/Telegram so that no news are lost, but duplicates are removed and bad writing is reformatted.
Many of the things that get to deal with unstructured human inputs. Translation, summarization, data extraction, etc. The choice isn't between "reliable code" and "unreliable LLM", but rather "unreliable LLM" vs. nothing at all.
But I think more important aspect is that LLMs are getting quite good at coding - meaning they can be used to automate all the tasks that you could automate with scripting, but it wouldn't be simple, or you don't know the relevant tooling well enough, or you just can't be arsed to do it yourself.
For example, I've recently been using GPT-4 to automate little things in my Emacs. GPT-4 actually sucks at writing Emacs Lisp, but even having to correct it myself is much less effort than writing those bits from scratch, as the LLM saves me the step of figuring out how to do it. This is enough to make a difference between automating and not automating those little things.
It all seems to be correlated with growth - as in, the fastest-growing areas of the industry, ones that are hot and considered a career gateway, attracting the largest amount of outsiders - are the biggest circuses on wheels.
It's already happening. An MIT study came out last week that found that Amazon Mechanical Turk workers hired to do RLHF type training of models were using ChatGPT to select the best answer. And the web being polluted by AI generated content which then gets scraped into Common Crawl and other training data sets has been an issue for a couple of years now.
GPT and LLMs will allow good and seasoned programmers to produce better code in time-constrained environments.
I think you’re completely wrong about new devs producing worse code with this new tool. On the contrary, they’re going to be able to learn things it took you years to master in a matter of months since they now have a private tutor/mentor/reviewer/domain expert/consultant on call 24/7 for $20 a month.
As for using generated content as input, Microsoft has published a few papers showing that using curated generated content can be used to train specialized models that are more competent in their domain than the original models and the kicker is they didn’t even use humans to curate the content, they just used existing language models!
For example: Evolution through Large Models - https://arxiv.org/abs/2206.08896
We wanted to digitize them, but after using google Text ocr service, we couldn't get enough quality.
Using GPT I was able to fix typos and output everything as Markdown in a structured way.
With that, we created a nice Google doc .
Now I'm using DallE to generate suitable pics for each recipe to add to the book.
Its actually coming up quite good IMHO. We hope to give it as a present to my grandma on her 100 year birthday in a couple of weeks.
Would it be sufficient to have 1B, 3B or 7B parameters to achieve this? Or is it doable with 100M or even fewer parameters? I mean vocabulary size might be quite small, max context size could also be limited to 256 or 512 tokens. Is there any paper on that maybe?
> know how and where to look for answers
The point of calling them Agents is that they don't know how. All the examples, including AutoGPT that makes AI influencers go OMG AGI, are operating in a discrete action space with user-specified hints to select which action (or none at all).
"We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities."
They also trained a TinyStories-Instruct instruction following variant. It looks like the 28M parameter 8 layer model had a 9/10 on grammar and consistency. You could probably combine that with something like Jsonformer or parserLLM to enforce valid formatting.
So 512 is really inadequate unless you just want to make search queries using natural language.
Also I think current LLMs might not fit well for agent use cases in mid to long term because the RL they go through is based on input-best output methods whereas the intelligence that you need in agents is more around how to build an algorithm to achieve an objective on the fly - this requires perhaps new type of large models ( Large Agent Models ? ) which are trained using RLfD ( Reinforcement Learning from demonstration )
Also I think one of the key missing piece is a highly configurable software middle ware between Intelligence ( LLMs ), Memory ( Vector Dbs ~LTMs, STMs ), Tools and workflows across every iteration. Current agent core loop to find next best action is too simplistic. For example if core self prompting loop or iteration of an agent can be configured for the use case in hand. Eg for BabyAGI, every iteration goes through workflow of Plan, Prioritize and Execute or in AutoGPT it finds the next best action based on LTM/STM, or GPTEngineer it is to write specs > write tests > write code. Now for dev infra monitoring agent this workflow might be totally different - it would look like consume logs from different tools like Grafana, Splunk, APMs > See if it doesnt have an anomaly > if it has an anomaly then take human input for feedback. Every use case in real world has it's own workflow and current construct of agent frameworks have this thing hard coded in base prompt. In SuperAGI( https://superagi.com) ( disclaimer : Im creator of it ), core iteration workflow of agent can be defined as part of agent provisioning.
Another missing piece is notion of Knowledge. Agents currently depend entirely upon knowledge of LLMs or search results to execute on tasks, but if a specialised knowledge set is plugged to an agent, it performs significantly better.
I get that there are different philosophies on AI risk, but this is like reading an in-depth discussion about potential atomic bombs in 1942, with no mention of the fact that such a bomb could potentially level cities and kill millions.
If a field of research ever needed oversight and regulation, this is it. I'm not convinced that would solve the problem, but allowing this kind of rushing forward to continue is madness.
Like does this mean that it will completely obsolete the works of boffins like simon peyton jones and those other esoteric languages and everyone will talk to their computers in english language and the ones that will be the most effective computer enjoyers will be the high emotional intelligence salesmen? were they going in wrong direction for so long on purpose because they didn't want to awaken the real power of the computers for ethical reasons, or were they trying to do this the whole time but they weren't smart enough to figure it out?
The article talks about LLMs being hooked up to the Internet. Big difference.
If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.
1. The LLM needs to find an exploitable bug in a popular code base.
2. The LLM needs to write a reliable exploit for that bug.
3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system.
4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!).
5. The LLM needs to understand the control protocols of their industrial control systems.
6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks).
7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions.
What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close.
Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance.
So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output.
Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late.
As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
I wouldn't be surprised if this led to the death of 10% of the global population in about 5 years. That's 800 million people, or around 4000 times the combined death toll of the Hiroshima and Nagasaki bombings.
The Internet is the backbone of the modern world. Short of a global nuclear war, it's hard to imagine a more catastrophic event than it suddenly becoming unavailable.
The loss of many of these records in the financial and governance would certainly lead to a lot of dollar losses. But dollars aren't people.
As for economic consequences, millions of jobs would just no longer make sense, and many others would get harder and frequently much less efficient. Certain major categories of product no longer make as much sense; what's an iPhone that can't connect to the internet worth? It's just a telephone with a camera (that takes images you can't share).
It could be deadly to some degree? All it takes is for the power to go out somewhere exceedingly hot in July, as the loss of air conditioning can be deadly for the elderly. But as deadly as the nukes the US dropped on Japan? I don't see it.
Nope. It's just a camera. Because the "telephone" part relies on infrastructure that is unmaintainable without the Internet, and would probably stop functioning in a matter of days.
And boom, you're back to the 19th century, where telephones weren't a thing. Except that unlike in the 19th century, there isn't any infrastructure to make things work in that situation.
How does the hospital order medical supplies now? By paper mail? Sorry, mail can't be delivered anymore, since all postal services depend on logistic systems that in turn depend on the Internet for coordination. Also, printed catalogs haven't existed for almost 2 decades, so the hospital won't even know what supplies are available.
And either way, the supplier doesn't have anything in stock, because their entire supply chain has collapsed, because international trade isn't a thing anymore. Did I mention that GPS (and thus most of sea/air navigation) has stopped working because the ground-based infrastructure needed to keep it running depended on, you guessed it, the Internet?
Still think that won't kill more people than the atomic bombings (which killed "only" 200k)?
My point on the iPhone was about what would happen after telephony infrastructure and supply chains had recovered (however long that takes). In the immediate aftermath of the shutdown mobile phones would absolutely lose the ability to make or receive calls. But again, I expect this to recover somewhat after a few months.
Your point on hospitals not being able to order supplies is a good one, regardless. Without blood supplies or dialysis equipment (or a bunch of other things), people would absolutely die. Perhaps those first weeks would be deadlier than I anticipated.
That's trivial in the industry context. A chunk of the stack is running decade old stuff.
> 2. The LLM needs to write a reliable exploit for that bug.
That's what Metasploit is for, isn't it?
> 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system.
See 2. if that's your strategy, but there are others. Such as, plant operations staff using random LLMs-as-a-service in their work (despite corporate saying not to do that; but it's not like the bosses don't do it either).
> 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!).
Not hard at GPT-4 level, will only be easier. If it starts with a goal of doing something bad at scale, it will recognize a mayonnaise plant as an eligible approach, should it "cross its mind".
> 5. The LLM needs to understand the control protocols of their industrial control systems.
All documented and already part of the training set. I know that one for a fact, because I've been "chatting" with GPT-4 about some nuances of industrial protocol, and getting it to write me example code.
Industrial stuff may be closed-source and expensive, but the documentation and specs and marketing blurbs are to be found publicly. Most underlying protocols are public and well-documented (ish). The recent push for IIoT / "Industry 4.0" is actually trying to replace most of that proprietary stuff, secure by obscurity, with web-adjacent tech - exactly the thing that LLMs know best, because out sheer openness and popularity of everything webshit.
> 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks).
I bet that, should you work around "I'm sorry, as a large language model trained by Open AI, I'm afraid I can't do that" issue, GPT-4 will happily give you 5+ different ways to make mayonnaise lethal. It's not rocket science - it's industrial food production. A chunk of the processes there exist to ensure the product won't develop chemical or bacterial contamination.
> 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions.
Nah, it just needs to spoof some PLC outputs somewhere, or a data feed that goes to the model-predictive control. There's a risk of triggering alarms somewhere, and hopefully most of the naive approaches will get caught in the late lab testing / QC stages, but still - you can get far without triggering anything but maybe a dashboard warning about some outlier values, that plant operators brush off as more bugs in the industrial software.
That's if your goal is to weaponize mayonnaise. If you want to blow up the plant, well... skip step 7.
> There are a few billion dollar businesses in those steps.
If you saw how some of those businesses work, you'd be surprised we're all still alive.
> walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
Nobody is saying that GPT-4 can do it on its own. But to the extent that GPT-4 or a model more advanced than it already captures some essence of generalized thinking, and given the creativity people are showing in constructing increasingly complicated chains of LLMs and classical tools to extend both the breadth and the precision of that generalized thinking ability, plus giving them every possible tool in the world, it's not hard to imagine those systems getting capable enough to screw stuff at scale.
The ultimate argument is that atomic bombs, bioweapons and even climate crisis were all done thanks to intelligent agents. Intelligence is what gives rise to those threats, so by itself, it's more dangerous than all of them.
Also:
> What you're vaguely hinting at is extremely high skilled labor. (...) I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close.
We've only been dealing with AI models capable of basic coding tasks for less than a year. We've barely even begun to apply optimization pressure to this capability. So even as LLMs are "nowhere close" today, I wouldn't take the bet that they will remain "nowhere close" a year from now - there's absurd amount of money and interest invested into making them capable of this, by proxy of making them capable of software dev, or high-level thinking in general.
When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise.
Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs.
Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).
Yes to both, sadly.
> Because there’s no way that someone will use one to control the automated processes in a factory.
What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you.
> It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.
The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.
That's the worst part of it.
Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows.
But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting.
(Hell, it's the first thing in tech in the last decade, that I find exciting.)
--
[0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.
I know it's technically crazy sounding, but for all we know, Earth is the only place with biological life, I think it's better to flip the coin somewhere else if that's what you want to do. Don't screw up here.
Yes, maybe the AI will come for Earth, but I feel like there's less chance of shit going really bad with some distance.
The doomsday AI cultists didn't come up with any of these ideas. The Matrix was a hugely popular movie in 1999.
chatGPT doesn't make the person who has believed since 1999 that AI is going to turn us all into Duracell batteries more correct. It is the same basic thought with with paper clips replacing batteries.
ah yes, the only people consuming such things should be The Big Corporations and Big Governments?
---
Snark aside (Sorry, I'm just a bit tired of people having a kneejerk reaction to this without even thinking),
we're talking about autonomous agents. Huginn and the like have existed for years. If they can be even somewhat better automated with NLP and the sort, it'd be excellent instead of having to actually manage and configure/code individual agents. and Guess what, these agents have ALREADY been connected to the internet. For years.
Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
> all the computers
The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct.
Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
Can you always tell if it's doing bad?
Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep?
When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave?
Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?
See https://youtu.be/ld-AKg9-xpM?t=30 for a counterpoint.
At any rate, if the AI is smart then there are alternatives to begging.
It would be poorly run because pure money is not a strong enough incentive to hire someone to commit a crime for you, because that money can easily disappear once it is seized by the local government. Usually criminal organizations operate on a degree of trust, trust that would be hard to establish from some LLM that can, at best, fake a face on a zoom call.
Satoshi's coins are easy to play around with because they are at least legal to hold and sell (for now). If they weren't, few would bother with them.
Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident?
How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves?
That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).
But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice.
"Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it?
What is to AI, that which global warming is to coal power?
What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities?
We organised ourselves to build the latter, but we still haven't globally dealt with the former.
Also, you’re wrong about flushing toilets necessitating sewers. They were still manually emptying toilets in London when they built its sewage system. So sewage systems became popular before flushing toilets were widely adopted.
China building new coal plants (while also building new solar, wind, and nuclear) suggests that is not the only reason.
And it's not like any of the complaints about PV elsewhere are safety related.
> So sewage systems became popular before flushing toilets were widely adopted
What existed was not, however, sufficient. That is what necessitated building a massive new system:
"""The Great Stink was an event in Central London during July and August 1858 in which the hot weather exacerbated the smell of untreated human waste and industrial effluent that was present on the banks of the River Thames. The problem had been mounting for some years, with an ageing and inadequate sewer system that emptied directly into the Thames."""
"""During the early 19th century improvements had been undertaken in the supply of water to Londoners, and by 1858 many of the city's medieval wooden water pipes were being replaced with iron ones. This, combined with the introduction of flushing toilets and the rising of the city's population from just under one million to three million,[b] led to more water being flushed into the sewers, along with the associated effluent."""
"""The Building Act 1844 had ensured that all new buildings had to be connected to a sewer, not a cesspool, and the commission set about connecting cesspools to sewers, or removing them altogether."""
Just because a thing existed, doesn't mean they had previously done it at scale: the Ancient Greeks had electricity, railways, and steam; battery cars predate gasoline; we've today got the tech for a fully global power grid; none of this is at the scale relevant to the problems, and the same can be assumed for at least one possible thing that AI might force us to consider.
My guess would be spam and filters. Already exists, but likely to become so much more severe the old solutions will no longer hold.
Not that it really matters to my core point: what's the AI equivalent.
Spam and filters are a great example because spam filters are already AI. Literally neural networks trained to recognize spam.
LLMs are great at classification too so once they are small and cheap enough they could certainly be used as spam filters.
These videos changed my mind entirely about AI risk.
I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments.
Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end".
He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.
If that's your goal, what's dumb about achieving it?
I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning”
The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.
Oh please. Not everything has to be regulated-to-hells before a use case is even found on this. Autonomous agents have existed for decades.
If it can automate agents like huginn[0] with natural language, I'd be very happy. Autonomous agents doesn't mean it's going to take over the world autonomously. Let's lower the fearmongering a bit.
The rules governing LLMs, while in principle mechanical, cannot be accurately controlled or even understood by humans. They are "autonomous" in the sense of animals, which may be trained to some degree, but may still surprise or even become a danger to their "owners", no matter how much effort is spent trying to control them.
An LLM may do what you ask it to, using the methods you expect it to use. Or it may do what you ask it to, using methods you weren't expecting at all. Or it may not do what you have asked at all. This isn't comparable to a rule-based system like Huginn.
>They are "autonomous" in the sense of animals
That's a really weird equivalency, which frankly, I'm not even sure is true.
A mind that's a black box to us, that we can predict to a degree based only on observing it at work, that's also a general-purpose intelligence we're trying to employ for limited set of tasks. It's not a bad analogy. Like animals, LLMs too have the capacity to "think outside the box", where the box is what you'd consider the scope of the task.
The term "AGI" for many has morphed into meaning something that's like simulation of a person, but with godlike infinite IQ and powers. So it's a special type of ASI now. Not my definition, just the sloppy way most people seem to think about it these days.
So for those people, to get something dangerous, we have to create a full emulation of a person , give it a 1000 IQ, maybe make it operate at 1 million times human speed, etc.
But the reality is that for a group of AI agents to be dangerous they just need to be very effective problem solvers connected to the world.
I will say that the idea that many AI agents will not be connected to the internet or embodied is unrealistic. But I do think it's critical as you point out that we have awareness of the dangerous of full autonomy and effectors.
The trick is that right now we have a very strong economic motive to increase the performance and efficiency of LLMs. And I believe that we should because there is huge upside and the state of the world is horrendous.
But within only a few years (certainly less than ten) of optimizing this, we will be walking a fine line between tools that are extraordinarily empowering on the one side and completely out of our control on the other.
All we need for GPT to be dangerous is for it to get maybe 25% smarter and much faster. We should anticipate 50-100 faster than humans output within a few years. Within 5-10 years those systems will be extremely affordable and widely deployed.
The danger starts to emerge because the the extreme performance difference between human and AI. You can't tell the AI to wait for you to make decisions, because while it is waiting a few hours for you to figure out what it's talking about, the equivalent of days worth of activity has been accomplished by your competitor's AI agent.
For people to see the danger it could help to explain the exponential track record of computing efficiency performance and also create demonstrations of what it's like to compete with something that thinks 50 times faster than you. One would think that people could extrapolate from video games or something. But I guess the problem is the open world video game agents are still largely unintelligent so it's hard to imagine.
Autonomous agents need guardrails and oversight. An autonomous agent let loose with all the tools in the world will in essence lead to an outcome which is not predicted to be in our favour.
Which is why the Open AI app store and plugins scare me more than anything else - more likely than not they are tool and data feeders into a large scale autonomous system.
Given the current state of the art, it won't lead to any worse outcomes than a somewhat mentally impaired human "let loose on the world". And since training scales quadratically (doubling model size means double the amount of data is needed to train it optimally), and we're already at the limits of current hardware, that's unlikely to change much any time soon.
In a way GPT-4 is like a mentally impaired human but in other ways it's superhuman. It operates faster than a human in many contexts. It has vastly greater knowledge. Agents based on LLMs could communicate and process new information practically instantaneously.
And the important point that people are in denial about is that GPT-4 reasons effectively. It's far from perfect and has some strange failure modes, but demonstrates the potential of these systems.
It's not accurate to think that LLM performance can't be improved without doubling size. There are many approaches to efficiency recently demonstrated that don't require larger datasets.
We now have many geniuses with billions and billions behind them pushing hard to optimize this specific application from all directions. Modifying the software that runs the model, the model parameters, model architecture, and the hardware. In particular for hardware there is now a large increase in attention to novel compute-in-memory paradigms or techniques.
GPT-X will have at least 33% higher IQ and 50-100 times faster output within no more than 5-10 years. Quite possibly less than that. Humans will not be able to compete with that. The only option will be to deploy their own AI agents.
And there will be a strong incentive to increase the level of autonomy for the agents, since making them wait for human input a few hours means that the competitors' agents race ahead doing the equivalent of days of work in that time frame.
This delegation of control to agents with superior reasoning ability sets the stage for real danger. Especially in a military or industrial context.
The internet will fill up with bots even more than it already is. Maybe it will be nicer, because smarter bots would be more fun to talk to.
The fact that hacker news can read this—which is what I’ve been afraid of since a few weeks into DALLE2–and still not see the coming wave, is more than a bit funny to me.
Also scary.
Mostly scary…
So imagine that world but with garbage coming out 100x faster as it can tailor political talking points down to a fractional percentage point of voters. Oh your car was stolen by a minority? Here’s an ad that mentions car thefts, going beyond the prior ad of just generic minority-hate.
There are a number of systems where someone bolted together components like this. Now we need more demos and evaluations of them. I just read a comment about someone who tried a sales chatbot. It would make up nonexistent products to respond to customer requests.
The underlying LLM systems need some kind of confidence metric output.
Right…sounds quite reckless?