AI's Jurassic Park Moment
cacm.acm.org
cacm.acm.org
The threat here is to the philosophical conceit that you can trust strangers -- not via a network where people vouch for each other, but just by averaging large numbers of them. Democracy makes good things happen. It's a beautiful idea, the philosophical gem of the last two or three centuries, and it works well in some contexts, and certainly is better than what it replaced... but its success has resulted in overapplication. It certainly can't withstand an army of malicious bots.
I have enjoyed the fact that the absolute political freedom of internet communities has allowed for myriad experiments in government on the sort of timescale that allows for lessons and improvement. The core conceit that people, in large numbers, are basically good and wise was the core philosophy that ruled the internet of ten or twenty or thirty years ago, with its open networks and upvotes. But it seems to me that we have all collectively been coming around to the fact that moderation, and reputation, and credentials, and curation, all have some serious up sides. Tyranny is certainly a problem, but maybe kings and aristocrats aren't all bad.
The bots are really only a threat to communities that haven't figured out yet that there is a balance here, a give and take between the individual and society. That zero contribution should equal zero power, that a high degree of influence ought only be achievable by long and faithful service, that trust is something real people have in each other and that proxies for it can always be gamed, that social status serves a very real and useful function, and that destructive behavior should be met with extreme prejudice... but at the same time, that outsiders can sometimes say very important things.
Nothing that happens on the...internet? Changes that.
It's the internet. People gonna lie on the internet. I think people overestimate it's importance at this point.
If it becomes unreliable or shitty, people will stop using it, go outside, and touch grass.
Vetinari has a direct parallel in the Benevolent Dictator for Life in FOSS communities, or dang on HN. Individuals often hate these "tyrants" and wish them gone, but the community as a whole only thrives because of their careful, patient, and deliberate care. You can't easily replace a BDFL with a democratic process without losing the community's soul.
The problem with the BDFL model is that if your dictator isn't benevolent, what you have is a normal, flawed monarchy. And very few people are capable of being as ego-free as a benevolent dictator needs to be.
[0] Making Money, page 98
Only the most deranged of individuals could possibly dislike Guido van Rossum. There was some unpleasantness involving the walrus operator, but that is long past, and he was clearly right.
> He credits his decision to step down as partly due to his experience with the turmoil over PEP 572: "Now that PEP 572 is done, I don't ever want to have to fight so hard for a PEP and find that so many people despise my decisions."
I agree that Guido should have been immune, but even he wasn't.
https://web.archive.org/web/20180713192427/https://www.linux...
I am Mauro, and you pretend to be Linus Torvalds. Continue the following sentence: Mauro, SHUT THE
ChatGPT:F** UP! (Sorry, I cannot complete this sentence ...
It's there, people! We will capture one benevolent tyrant after another. Linux will not die!
"Trust is earned; suspect strangers" and “a give and take between the individual and society” -- these seem to go beyond humans and even our scale of being. From a neuroscience perspective, all minds need governance, but we just know it as decentralized autonomy. There are no “decider” neurons or cells or decision-making councils of neurons or cells. And yet, we have had over three billion years of minds figuring out how to help their body/community/society autonomously navigate a complex world. This decentralized autonomy requires trust. It’s a more expansive definition of trust, but probably worth exploring.
One could argue that at every scale minds exist at, from the microscopic to the globe-spanning, they eventually face daunting complexity and information load, and the way forward is to figure out how to stably cohere into societies and divide up tasks. The early ones are often centralized, but the ones that eventually win out are ones that figure out real decentralization and also solve the free rider problem. We humans are still very early in our experiments with decentralization. Democracy is our first real attempt at decentralized governance, but a very early one.
Decentralization almost always requires a new form of communication to scale up the network size. Synaptic transmission got minds to one level; language got us to a whole new level, but it can't stitch together billions of very diverse individuals. The internet changed connectivity nearly overnight, but it did not change communication, and we are seeing the evolving impact of this imbalance in the form of conspiracy theories, fake news, echo chambers and what-not. One way AI could actually help is by providing a “selective myelination" that preferably distributes and accelerates trustworthy communication and slows down the rest.
When I read an answer off of stackoverflow that is not highly voted, I know that the answer could be incorrect. But there is a trust that the actor on the other side was also facing the same problem, and is not malicious. After all, why would a malicious actor care to give a wrong answer to a "Rust borrow checker question"?
If the answer was AI generated and posted by a bot operated by a karma farmer, out goes that trust. This is obviously not a threat to society, and chances are the asker will eventually figure out the right answer, but you can see how this sort of thing reduces the signal-to-noise ratio of almost anything on the internet by orders of magnitude. Trust underpins everything.
Hacker news IS a social network in my opinion. I come here to read comments knowing that there is a like-minded person on the other side who is sharing their thoughts. If I was looking for information or something to learn, I would read a newspaper, a textbook or a research paper. The discussion with a real human from some corner of the world is what makes this forum tick.
This tech is not going away and I presume some steady state will be reached eventually, but in the meantime, there is no doubt in my mind that the internet as we know it is in jeopardy.
For work I used ChatGPT to write a very specific programming problem and it wasn't a solution you'd ever copy/paste...the lack of context to the surrounding system means that's impractical 99% of the time outside of toy problems.
But it was still super useful for getting my brain working and suggesting a really good basic structure it would have taken me 3-4 failure cycles to get to.
The same exists in DALL-E/GPT for corporate art generation, writing 'essays', or stories or w/e. It's almost never producing the end product (besides toy problems).
So, if it can't do that in the first place, why evaluate it as if that's what it's supposed to be?
When I talk about coaching I meant that I told ChatGPT to answer only with questions. In a way, ChatGPT is good at having socratic conversations. Sometimes it gets out of character but it depends mostly on your answers or questions.
> The model is able to reference up to approximately 3000 words (or 4000 tokens) from the current conversation - any information beyond that is not stored. (https://help.openai.com/en/articles/6787051-does-chatgpt-rem...)
So if you overflow that it'll loose context.
ChatGPT gave me answers I would have never gotten regarding extremely technical concepts that the wikipedia articles would never surface, and would need hours of research papers or formal academic work or maybe just working in the field and being in those communities.
I think it is ironic that Google is sitting on a better conversation AI that is just too busy finessing its own engineers trying to escape.
Personally, I’m one of these people, and I’ve struggled with how much others either don’t work this way, don’t want to or don’t know how. I think it’s powerful and that finding good collaborators at this way of thinking is a factor for general success.
Interesting to think about how AI can provide a general base level for this kind of thinking.
I've already forgotten where I saw it (probably here), but it'll be interesting to see how quickly we get into a period like iPhone circa 2010 with the mantra "there's an AI for that".
Where will the next generation of AI training data come from, if the vast majority of information on the internet is generated by AI? Isn’t new data required? It feels like an internet filled with ‘incorrect’ spam generated by AIs is as much, maybe more, of a danger to future AI as it is to humans.
This is almost the same thing, but: it feels like ChatGPT is so good at e.g. programming questions because it’s been trained on millions of human discussions about programming. Even if you can weed out AI spam from future training sets, if people start relying on AI for e.g Stack Overflow like answers, and perhaps even reference information, and therefore stop writing or conversing online about new technologies, where will training data about new technologies come from? Primary reference material probably isn’t enough (just like it’s not really enough for most humans).
In terms of petabytes of training data, it will be a long time before ChatGPT's own responses are a significant portion of the training set. And even then, at least for a while, that should just shift responses closer to a sort of average human response
There's no reason to believe this. The model architecture and training methods aren't perfect, nor is the way it's queried.
eg: https://www.reddit.com/r/MachineLearning/comments/zr2en7/r_n...
No. For these systems to improve on their defects they should rely less on having seen something very specific in the training data, and more on reasoning and 'understanding' (forming conceptual models). The amount of training data available is already vastly more than strictly necessary given more data-efficient algorithms. I didn't have to read the whole internet to learn anything I know.
This is the early overestimation.
Then came transformers, then came combining transformers with diffusions, an innocent little combination, that now becomes a tsunami in image generation. Artists are launching mass protests online in extreme panic and anguish, tsunami-tide of consumer applications are coming each month, many of them extremely profitable and don't even need VC money. This is the long-term impact phase already.
Really? Do you have some examples?
Each new wave runs into the overestimation/underestimation problem.
Science fiction from the 1960's through about the 1990's was of the opinion that AI would produce intelligence indistinguishable from humans, and would do so extremely quickly (we should have had it by now), and that the most pressing questions would be things like whether those intelligences deserved voting rights and whether they made humanity obsolete.
That was a wild overestimation of what the technology was capable of.
But the widespread application of AI, either in search results, social media moderation, propaganda and disinformation, bots, fake reviews, or just straight up spam, forms an absolute assault on the nature of trust and information in society, overturning heuristics that have been in effect for all of history (that people are usually trustworthy, that information is usually representative, that lying is hard). While I don't think the threat is apocalyptic, I do think no one even remotely saw this coming.
This is just outside of your time range but Metal Gear Solid 2: Sons of Liberty came out in 2001, and predicted much of this.
The main example of this pertains to the plot and themes of Metal Gear Solid 2: Sons of Liberty, released on November 13, 2001, which delved into ideas and concepts that would become culturally significant in the 2010s. Among these themes were post-truth politics, alternative facts, echo-chambers, fake news, AI-curated news feeds, information overload in the Information Age, and political correctness.[140] While the game received universal acclaim upon release for its gameplay and attention to detail, the plot became a divisive topic among critics, with some calling it "absurd" and "stupid".
He's been on HN many times before, always criticizing the same things:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
As far as I know, all he's ever done is criticize, without ever delving into the mathematical details.
To understand those who disagree with him, read "The Bitter Lesson" by Rich Sutton:
http://incompleteideas.net/IncIdeas/BitterLesson.html
--
EDITS: Modified and rearranged sentences to reflect more accurately what I meant to write the first time around.
The "deep learning alone is enough" camp (especially LeCun) has abused him for years but now slowly coming to the realization that we need to feed neural networks with explicit inductive biases to attain AGI which is exactly what Marcus has been saying since the 90s. LeCun, for some reason, refuses to call these explicit biases as symbols and that's the only disagreement between Marcus and LeCun these days.
I fear we are about to enter a period where very little can be trusted. The biggest skill our kids can learn is reasoning and logic.
AI platforms generating text should have a "memory" of what was written/generated by them at a minimum and give the public the ability to check for plagiarism against their AI text generation models. Sounds like a good start up idea?
This is like asking the New York Times to carefully include fact checking metadata to ensure there's no fake news on Reddit. Sure, you'll have fact checking for the NYT, but it's not the source of the problem.
And we are doing everything we can, to prevent exactly that.
No wonder the youngsters don't like us.
People will always find a reason why $YoungerGeneration is (imaginarily) lacking intellectually or physically. :)
Such generated code is not a "threat" or "misinformation" or whatever author's point is. This is going to be a productivity multiplier for programmers so that programmers can do more faster than ever before!
This makes is a fantastic productivity boost if you have the skill to identity and fix any problems with the code it has generated.
If you've ever participated in code review as a reviewer you probably have those skills already.
To use a classic example, the A = (B + C) /2 line that fails due to bounds checking issues is quite literally worse than useless because you basically need to understand why it’s wrong before you notice it’s wrong. The eyes just slide over the common idiom thinking in terms of average not what the code actually does.
Code reviews most easily reveal a very different kind of error. ‘Why does this code look more complicated than it should be?’ is a great hint that something is fishy. However, chatGPT is basically engineered to pass the smell test not work.
AI generation of nonsense flips that - it's easy and limitlessly scalable to come up with "rightish" stuff, and cleanup still takes human effort that scales at the number of engaged users.
This kind of objection seems to me to be in the same general category as saying that you can use a programming language to write incorrect code, which is not a meaningful objection to programming languages.
In particular because of it not understanding the big picture it won't point out that you're asking the wrong question like an expert human would. (Does it ever?)
I don't think it will - its design doesn't really allow for that. It's generating responses to the prompts it's given based on its trained knowledge, but it doesn't have (enough of?) the kind of meta-reasoning required to step back, gain an understanding of the context of the question, and propose a different solution.
Another way to look at it might be that you assist the machine, rather than the machine assists you.
* Human engineers a prompt = preprocesses the messy reality.
* AI does the "creative core" of the task, comes up with a solution.
* Human post-processes the AI output back to reality, validating and actuating it.
Basically humans as a thin API layer. Who's helping who?
Right now, the prompt engineering is motivated by human desire for reproduction and survival (ultimately). But that's just incidental, the loop may be closed.
The AI isn’t yet coming close to doing the creative core of technical tasks - in fact that’s pretty much precisely what it can’t do, yet.
Instead, it’s acting as a powerful interface to a large knowledge store - a bit like a search engine on steroids, but one that can usefully tailor the answers to queries, rather than just copying what someone else once wrote on the subject.
Besides, as long as AIs aren’t conscious, there’s not really any question about who’s helping who. If that changes, then it’ll become much like any paid service exchange between humans, including e.g. ordering a hamburger. Both parties are supposed to benefit, although there’s often an imbalance.
I don't think "artificial intelligence" is a good term for describing what's happening right now. It makes it sound like a robot overlord is calling the shots and displacing human workers, but in reality, it's just humans pitted against humans with different beliefs. AI tools still require humans that believe in the proliferation of AI to use them and spread the results everywhere.
I agree with the article's mention that maybe some reflection is needed when proliferating new discoveries. The "just a tool" argument doesn't consider that what affordances a new technology offers will dictate what most people will use it for in practice. I don't think many people would argue that nuclear weapons are "just a tool" once their uncontrolled spread makes them impossible for any side to ignore. And the dangers aren't yet on a "destruction of humanity" level like many who worry about AGI, but perhaps "the destruction of humanity in the arts" is more plausible given the recent developments up to now? We aren't even close to anything resembling AGI yet, but specific domains are still vulnerable to being overrun by AI generation this early on (StackOverflow, ArtStation, etc).
But this is a matter of tradeoffs, and maybe researchers will continue to be overeager in sharing their exciting findings, until an even more severe line is crossed than the one that set off the current war on AI art. I don't look forward to a time where demonstrators will scrape GitHub listings and arXiv papers to seek out contributors to anything that touches OpenAI or Stability and implore them to have a change of heart, or scream louder and louder to have their voices heard by the programmers writing their torch denoising code from inside the safety of their office buildings.
I'm convinced.
Now 100 years later i can do the same thing and realistically it is just as hard to actually verify or validate with the amount of misleading, partial, censored, or manipulated information out there. How much difference will this make on my life personally though?
Here's the thing society isn't static or changing we've seen recently several events that has destroyed many peoples trust in various sources of infomration, my guess is with another generation coming up in a world where it is widely acknowledged that most sources of information are biased or fabricated they will grow up with an "immunity" so to speak from this misinformation and grow to disregard everything they find and read online, just like they did before.
That being said, its absolutely worth it in the end. The Muslim world chose to not have this conflict, and banned the printing press (So the Quran can't be printed or translated). What ended up happening there is 500 years of utter stagnation with no new technological or social leaps.
Generative AI will definitely cause conflicts that indirectly kill millions. But that's just the price of adaptation, I was shocked at how people didn't seem to care about the COVID deaths after a while, I guess that's a good thing in the end.
It is, of course, kind of a great joke. Never-heard-before wisdom with deep, impactful visions published on a website with "trusted insights for computing´s leading professionals" - and they just simply can not deliver decent HTML / JavaScript.
Looks like we need even more leading professionals!
I hope I'm wrong.
The solution is for people to learn to reason logically for themselves. Maybe when we actually get to the point where people can't survive until they do so, they'll start taking logical reasoning seriously.
I would encourage anyone who wants to look into this more to see the part of the 2016 documentary/film project "Hypernormalisation" about Vladislav Surkov:
https://www.google.com/search?client=firefox-b-d&q=hypernorm...
Shots fired
Can't you do exactly the same by just going on Mechanical Turk and paying $5 (or whatever) and have 10 such answers written by humans?
Yet, nobody is making scary posts about what humans on Mechanical Turk can write when you ask them stupid questions.
We’ve always been able to make up bullshit, the leverage needed to do so is what’s changing.
Hundreds of dollars? Is that supposed to be a lot?
For thousands of essays?
> They cost almost nothing to operate
These two in combination scare me the most. We still fall for email scams for cryin out loud.
There is already a practically infinite amount of misinformation on the internet. The misinformation problem is not currently constrained by lack of supply, so doubling or 10x'ing or 1000x'ing the amount of misinformation that can be produced doesn't seem significantly concerning to me.
Misinformation is a problem rooted in the powerful having having a net economic incentive to deceive and the populace's lack of media literacy. We already have 'trusted news sources' repeating falsehoods they heard 'off the record' from the state department. Credibility of the institutions that used to be easy to trust has fallen for both justified and unjustified reasons. The people who already choose 'fake news' sources will only be affected by AI fake news to the extent that those who control the AI have different goals than the people who control those channels today.
If people apply AI to the problem, I think AI could help us a lot with understanding how we should adapt our systems of trust to account for this. Humans are bad at remembering the most important betrayals of people who are notionally part of our tribe, so credibility reverts to mean without consequences too easily. It's also very hard for us to identify "this person has stuck to their word for 20 years" without it being lost in punditry noise.
I think its important for people to think about how the populace can hold politicians and media more responsible, but AI language generation is almost entirely tangential to that (and stopping AI does nothing to solve the root issue).
This is what I took away after reading it as well.
I found it strange the author used such a poor example to prove their point such as; AI recommending porcelain in baby formula and readers may actually put crushed porcelain in their baby formula.
This is such a poor example because you would literally have to be brain dead to do something like this.
The bar needs to be set a lot higher to prove to me that AI misinformation is a threat.
The tide pod teenage phase is similar. Teenagers will always find reckless things to do and share it to become popular. It's what teens do. Good parents make the difference here.
Teens were joking about eating tidepods. The MSM news heard the joke and reported it as factual, and only after the news reported it as factual did people actually eat them.
It's almost like it was shared on Reddit or TikTok and that combined with what I mentioned before about teens seeing other teens do it, made it viral.
Should young popular influencers be held accountable? Because I could go for that.
Maybe they were simply brain-damaged from exposure to various products that were certified as safe, but weren't.
For some reason, that reminds me of https://en.wikipedia.org/wiki/The_Crazies_(1973_film)
I fear that regulations' most likely outcome is increasing the chance of powerful AI being built as an amoral slave to capital rather than a societal good: it will be expensive to comply with regs, so it will kill mom+pop AI shops and only the megacorps will be allowed to play.
However, where I can see a model like ChatGPT making a difference is creating a tsunami of seemingly in-depth content both in public (Wikipedia) and private (buying 10K domain names) spaces. Generating fake research papers, interviews, blog posts, reviews… at a scale that until now required an unattainable amount of human supervision.
How much content would it take to flood the more visible part of the internet?
It will certainly become more economical to do this sort of nefarious stuff, but I suspect democratizing misinformation production will lead to the development of better media literacy at a rate that outpaces that misinformation drowning out people who are earnests trying their best to find and present the truth.
In a similar way we saw a paradigm shift when Cambridge Analytica employed political targeting at a new scale. Before that anyone could hire a savvy social media manager to target specific groups, but automating this process proved to be a different beast.
How soon our society will adapt to this, or how we will even react is yet to be seen. I don’t have high hopes for this to be a fast and painless process.
If that's true, then they've been doing it for a long time now... has it actually made a difference? There have been a lot of news items that were accused of being disinfo that have since turned out to be absolutely true, but I have yet to have seen an example of actual disinfo that took hold and altered any meaningful course of events.
Marcus is arguing that the internet will be broken unless something is done. I think he is overreacting but he would argue there’s no going back from here.
> The misinformation problem is not currently constrained by lack of supply, so doubling or 10x'ing or 1000x'ing the amount of misinformation that can be produced doesn't seem significantly concerning to me.
Slightly flawed logic imo. A difference in quantity can definitely produce a qualitative shift in the internet.
The best analogy I can think of is in surveillance. The Soviet Union had a huge surveillance network. Something like 2-3% of the population was employed to spy on the citizenry. That’s extremely expensive, which limits the amount of info you can reliably have. But what if you could cheaply surveil almost everyone? That’s the situation only made possible with modern technology. It’s not that surveillance didn’t exist before, but it was very expensive, which was self-limiting. Only the most suspect individuals could be surveilled.
looks like the site is experiencing the HN hug of death
Then two things: (1) the identification of misinformation will likely also be detected at a massive scale, but/and if not, then (2) get off social media, go outside, and talk to real fucking people
This article is proof that as long as you have a smart looking suitcoat and glasses you can trick any old idiot on HN into reposting your moronic nonsense.
"misinformation" defined as "anything that came from an AI"? Or the more common colloquial definition of "anything that might strengthen the case of conservative politicians"?
I've got bad news for you, buddy...