Jan Leike Resigns from OpenAI
twitter.com
twitter.com
While Jan's (very pithy) tweet was later in the evening, I was reading other posts yesterday at the time of Ilya's announcement saying that Jan was also leaving.
What many of us are actually experiencing is existential terror about capitalism itself, but we don't have the conceptual framework or vocabulary to describe it that way.
It's a cognitive shortcut to look for a definable villain to blame for our fear, and historically that's taken the form of antisemitism, anti-migrant, anti-homeless, even ironically anti-communist, and we see similar corrupted forms of blame in antivax and anti-globalist conspiracy thinking, from both the left and the right.
While there are genuine x-risk hazards from AI, it seems like a lot of the current fear is really a corrupted and misplaced fear of having zero control over the foreboding and implacable forces of capitalism itself.
AI is hypercapitalism and that is terrifying.
AI Doomerism is actually capitalist anxiety.
So I imagine that the result would be that industry devolved into the manufacturing of luxury products, in the style of the top-class products of ancient Rome.
That is the person who actually made the purchase.
SAME way.google search is now a steaming garbage.pile.
Now, we've have the co-leads of the super-alignment/safety team leaving too.
Certainly not a good look for OpenAI.
There really doesn't seem to be much of a mission left at OpenAI - they have a CEO giving off used car salesman vibes who recently mentioned considering allowing their AI to generate porn, and now releasing a flirty AI-girlfriend as his gift to humanity.
From what we've seen of OpenAI, it does seem that any dissenting opinions will be bulldozered by Altman's product and money making focus. The cult-like "me too" tweets that huge swathes of the staff periodically send seems highly indicative of the culture.
Anthropic does seem genuine about their safety-first agenda, and strategy to lead by example, and have been successful in getting others to follow their safe scaling principles, AI constitution (cf OpenAI's new "Model Spec") and open research into safety/interpretability. This emphasis on safe and steerable models seems to have given them an advantage in corporate adoption.
Either way, if it's enough to cause them both to think it's better to research outside of the opportunities and access to data that OpenAI provides, I don't see a scenario where this doesn't indicate a significant shift in OpenAI's commitment to superalignment research and safety. One hopes that, at the very least, Microsoft's interests in brand integrity incentivize at least some modicum of continued commitment to safety research.
Besides the infamous Tay there was that apparently un-aligned Wizard-2[or something like that] model from them which got released by mistake for about 12 hours
We can’t just drop papers on arxiv. There is no way running your own twitter, github, etc as a separate group allowed.
I checked fairly recently to see if the model was actually released again, it doesn’t seem to be; I find this telling.
Tay way trivially racist, but boy was Sydney a wacko.
super-alignment co-lead with Ilya (who resigned yesterday)
what is super alignment? [2]
> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027.
[1] https://jan.leike.name/ [2] https://openai.com/superalignment/
I'm guessing, but OpenAI probably wants to start monetizing, and doesn't feel like they are going to hit a superintelligence, not really. That may have been the goal originally.
To some of us, that sounds like, "Fire all the climate scientists because they are needless cost center distracting us from the noble goal of burning as much fossil fuel as possible."
This is a tortured analogy, but what I'm getting at is, if OpenAI is no longer pursuing AGI/superintelligence, it doesn't need an expensive superintelligence alignment team.
Also, even if they never produce a superintelligence, they are likely to produce insights that would make it easier for other teams to produce a superintelligence. (Since employees are free to leave OpenAI and join some other team, there is no practical way to prevent the flow of insights out of OpenAI.)
An interactive mathematical model is not going to run away on its own without some very deliberate steps to take it in that direction.
What leads you to believe that's true?
There's zero chance LLMs lead to AGI or superintelligence, so if that's all OpenAI is going to focus on for the next ~5 years, a group related to superintelligence alignment is unnecessary.
Do you have a better analogy? I'd like to hear more about how ML models can't be intelligent, if you don't mind.
I'm pretty skeptical of the idea that we know enough at this point to make that claim definitively.
>A language model is a mathematical construct
That is like telling someone from the Middle Ages that a gun is merely an assemblage of metal parts not too different from the horseshoes and cast-iron nails produced by your village blacksmith and consequently it is safe to give a child a loaded gun.
ADDED. Actually a better response (because it does not rely on an analogy) is to point out that none of the people who are upset over the possibility that most of the benefits of AI might accrue to a few tech titans and billionaires would be in the least bit re-assured by being told that an AI model is just a mathematical construct.
You mean like PALM-E? https://palm-e.github.io/
Embodiment is the easy part.
Super intelligent AI seems contrary to the goals of consumerist Capitalism, but maybe I'm just not smart enough to see the play there.
Can somebody translate this to human?
If AI did run away and do it's own thing (seems super unlikely) it's probably a crapshoot as to whether what it does is worse than the environmental apocalypse we live in where the rich continue to get richer and the poor poorer.
Which we don't.
So we're not aligning it with corporate boards yet, though not for lack of trying.
(While LLMs are not directly agents, they are easy enough to turn into agents, and there's plenty of people willing to do that and disregard any concerns about the wisdom of this).
So yes, the crapshoot is exactly what everyone in AI alignment is trying to prevent.
(There's also, confusingly, "AI safety", which includes alignment but also covers things like misuse, social responsibility, and so on)
And for non-robotic AI, also flash-crashes on the stock market and that thing with Amazon book pricing bots caught up in a reactive cycle that drove up prices for a book they didn't have.
This is what most people mean when they say "run away", i.e. the machine behaves in a surreptitious way to do things it was never designed to do, not a catastrophic failure that causes harm because the AI did not perform reliably.
When people are not paying attention, they're just as dead if it's Therac-25 or Thule airforce base early warning radar or an actual paperclipper.
That's why Private Foundations are more popular than Public Charities even though both are 501c3 organizations, because they don't need to provide transparency into their operations.
It's one hell of a problem.
I agree it's a problem but it isn't incumbent on the 'x' peers to solve it. The burden of that goes to any supposed '10x'.
Long version:
I agree with you, though I would add that a superintellect at '10x' that couldn't look at the 'x' baseline of those around it and navigate that in an effective way (in other words, couldn't organize its thoughts and present them in a safe or good seeming way), is just plain not going to ever function at a '10x' level sustainably in an ecosystem full of normal 'x' peers.
I think the whole point of Stranger in a Strange Land is about this. The Martian is (generally) not only completely ascendant, he's also incredibly effective at leveraging his ascendancy. Repeatedly, characters who find him abhorrent at a distance chill out as they begin to grok him.
The reality is that this is an ecosystem of normal 'x' peers and the '10x', as the abnormality, needs to have "functional and effective in an ecosystem of 'x' peers" as part of its core skill set, or else none of us (not even the '10x' itself) can never recognize or utilize its supposed '10x' capacity.
For reference, I think most of us are '10x' in a particular field and that is our talent. Society-in-scarcity rewards talents unequally so we get status and ego resulting in a host of dark patterns. I think AI can ease scarcity so I keep betting on this horse for solving the real problem, which is ego.
> To me, the local minima looked "good"
AI's entire business [0] is generating high quality digital content for free, but we've never ever ever needed help "generating content". For millennia we've sung songs and told stories, and we were happy with the media the entire time. If we'd never invented Tivo we'd be completely happy with linear TV. If we'd never invented TV we'd be completely happy with the radio. If we'd never invented the the CD we'd be completely happy with tapes. At every local minima of media, humanity has been super satisfied. Even if it were a problem, it's nowhere near the top of the list. We don't need more AI-generated news articles, music, movies, photos, illustrations, websites, instant summaries of research papers, (very very bad) singing. No one's looking around saying, "God there's just not enough pictures of fake waves crashing against a fake cliff". We need help with stuff like diseases and climate change. We need to figure out fusion, and it would be pretty cool if we could build the replicator (I am absolutely serious about the replicator). I remember a quote from long ago, someone saying something like, "it's lamentable that the greatest minds of my generation are focused 100% on getting more eyeballs on more ads". Well, here we are again (still?).
So why do we get wave after wave of companies doing this? Advances in this area are insanely popular and create instant dissatisfaction with the status quo. Suddenly radio is what your parents listened to, fast-forwarding a cassette is super tedious, not having instant access to every episode of every show feels deeply limiting, etc. There's tremendous profits to be had here.
You might be thinking, "here we go again, another 'capitalism just exploits humanity's bugs' rant", which of course I always have at the ready, but I want to make a different point here. For a while now the rich world has been _OK_. We reached an equilibrium where our agonies are almost purely aesthetic: "what kind of company do I want to work for", "what's the best air quality monitor", "should I buy a Framework on a lark and support a company doing something I believe in or do the obvious thing and buy an MBP", "how can I justify buying the biggest lawnmower possible", etc. Barring some big dips we've been here since the 80s, and now our culture just gasps from one "this changes everything" cigarette to the next. Is it Atari? Is it Capcom? Is it IMAX? Is it the Unreal Engine? Is it Instagram? Is it AI? Is it the Internet? Is it smartphones? Is it Web 2.0? Is it self-driving cars? Is it crypto? Is it the Metaverse and AR/VR headsets? I think us in the know wince whenever people make the leap from crypto to AI and say it's just the latest Silicon Valley scam--it's definitely not the same. But the truth in that comparison is that it is just the next fix, we the dealers and American culture the junkies in a codependent catastrophe of trillions wasted when like, HTML4 was absolutely fine. Flip phones, email, 1080p, all totally fine.
There is peace in realizing you have enough [1]. There is beauty and discovery in doing things that, sure, AI could do, but you can also do. There is joy in other humans. People listening to Hall & Oates on Walkmans teaching kids Spanish were just as happy (actually, probably a lot happier) as you are, and assuredly happier than you will be in a Wall-E future where 90% of your interactions are with an AI because no human wants to interact with any other human, and we've all decided we're too good to make food for each other or teach each other's kids algebra. It is miserable, the absolute definition of misery: in a mad craze to maximize our joy we have imprisoned ourselves in a joyless, desolate digital wasteland full of everything we can imagine, and nothing we actually want.
[0]: I'm sure there's infinite use cases people can come up with where AI isn't just generating a six fingered girlfriend that tricks you into loving her and occasionally tells you how great you would look in adidas Sambas. These are all more cases where tech wants humanity to adapt to the thing it built (cf. self-driving cars) rather than build a thing useful to humanity now. A good example is language learning: we don't have enough language tutors, so we'll close the gap with AI. Except teaching is a beautiful, unique, enriching experience, and the only reason we don't have enough teachers is that we treat them like dirt. It would have been better to spend the billions we spent on AI training more teachers and paying them more money. Etc. etc. etc.
[1]: https://www.themarginalian.org/2014/01/16/kurt-vonnegut-joe-...
Teachers are hopefully empowered by AI to better adapt to the needs of the students.
I'd like to tack onto your mention of teaching. I have found teaching really pushes me to understand the subject. It would be sad to lose this ability to have "real" teachers, if everything goes to AI.
abortion
animal cruelty laws/veganism/vegetarianism
affirmative action
climate change(denial)
These are legitimate issues, but it is also totally possible to just "ignore" them and pretend like they don't exist.
However the paranoid part of me says that these crises and wars are just for the sake of letting people continue to ignore the truly difficult questions.
You mean like injection locking with oscillators? Or is this a new term in the tweetosphere
I mean it hides nuance in conversation.
https://openai.com/index/introducing-superalignment/
"Our goal is to solve the core technical challenges of superintelligence alignment in four years.
While this is an incredibly ambitious goal and we’re not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:C There are many ideas that have shown promise in preliminary experiments, we have increasingly useful metrics for progress, and we can use today’s models to study many of these problems empirically.
Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company."
Are you not aware of how many billions are getting spent on fake girlfriends on Only Fans, with millions of people chatting away with low paid labor across an ocean pretending to be an american girl? This is just reducing costs, but the consumers are already wanting the product.
I'm not sure I get the outrage / puritanism. Adults being able to chat to a fake girlfriend if they want to seems super bland. There's way more stuff out there that's way wilder you can do online, potentially also exploiting the real person on the other side of the screen if they are trafficked or whatever. I don't have any mental issues (well, who knows? ha) and genuinely would try it the same way you try a new porn category every once in a while.
The net sum of all LLM B2C usecases (effectively Chatgpt) is competing with AI girlfriends for rank 1.
It isn't just huge. It is the most profitable usecase for gen AI.
"A goal of a system is what it does"
"The internet is for porn" https://youtu.be/LTJvdGcb7Fs?si=8H1OzeyG5XzU-Qe8
Different methods of therapy appear to be equally effective despite having theoretical foundations which are conflicting with each other. The common aspect between different therapies seems to be "having someone to talk to", so I'm inclined to believe that really is what's behind the success.
>
> Different methods of therapy appear to be equally effective despite having theoretical foundations which are conflicting with each other. The common aspect between different therapies seems to be "having someone to talk to", so I'm inclined to believe that really is what's behind the success.
Just because talking is the common trait, doesn't mean that that's evidence that that is all it is. Paying someone to help you with the problem is also a common trait (and ironically, that is, no doubt, a contributory factor), but that isn't all that therapy is.
Let's say that there are three ways to solve a problem, and depending on context that we're not terribly good at determining, one of those ways will work quite often, one will work some of the time, and the other will be a disaster... but there's an equal probability that each of those ways are equally likely to fall in to each of those categories. Statistically, one could claim that how you solve the problem is not behind the success. In a sense, that would be correct, because the real determinant of success would be being lucky with the solution you chose to employ. While one could imply though that really it's nothing more than being lucky at choosing the solution, in reality without all of what's involved in that choice, the problem will remain.
Other kinds of conditions, like depression and anxiety, respond to a wider range of therapy styles. But those aren't the only conditions that people seek to treat through talk therapy. (And it's also an exaggeration to say that just having any conversation will help to treat anxiety and reopression. But it is probably true that treatment of these conditions is less technical and responds to a much wider range of styles.)
This isn't true. Different methods work better for different problems. I've been in behavior health for 7 years now. It's having someone with a lot of education to talk to, someone with education in social and psychological problems and healthy coping mechanisms.
If you are fortunate, you have people like that in your immediate circle, but increasingly few people do.
it's a bit more complicated than that
https://www.youtube.com/watch?v=Z37i8-FnAh8
and on top of this the method of therapy is to find better copings, not just to vent.
This is largely due to a crisis in the field in the late 70s/early 80s when several studies demonstrated that talk therapy had outcomes no different than no therapy. In both cases, some got better, some got worse, some didn't change. CBT was a direct result of that, prioritizing and tracking positive outcomes, and from CBT came a lot of different approaches, all similarly focussed on being demonstrably effective.
Talk therapy isn't a cure-all, but it's definitely more results-oriented than it was 50 years ago.
There's no defined stop point for physical development either... Top performing athletes still have trainers, and nobody sees that as a problem. If it's mental development though, it must have a stop point?
Meanwhile, the more therapy someone does, the more miserable they are compared to me. I’m the Usain Bolt of mental health compared to them. Makes me think their trainer is an idiot.
There is nothing stopping them from exiting therapy. The therapist may be aware that the person is still a basket case but if they are non-violent they are free to roam.
> If it's mental development though, it must have a stop point?
What is being developed, exactly?
- Patient choosing a specific mood problem/feeling they want to work on.
- A mood survey, where the patient rates their own level of e.g. anxiety, depression, fear, hopelessness. (e.g. out of 5 or 10).
- Therapy session, following his TEAMS CBT structure. Including patient choosing how much fear they'd like to feel (e.g. they want to keep a little bit of fear so they don't endanger themselves, but don't want to be overwhelmed by fear, 5% or 20%, say).
- A repeat of the mood survey, where the patient re-assesses themselves to see if anything has improved. There's no units on the measures because it's self-reported, the patient knows if the fear is unchanged, a little less, a lot less, almost gone, completely gone, and that's what matters.
That gives them feedback; if there is improvement within a session they know something in the session helped, if several sessions go by with no improvement they know it and can change things up and move away from those unhelpful approaches in future with other patients, and if there is good improvement - patient is self-reporting that they are no longer hopeless about their relationship status, or afraid of social situations, or depressed, to the level they want, then therapy can stop.
He's adamant that a single 2hr session is enough to make a significant change in many common mood disorders[2], and this "therapy needs to take 10 years" is a bad pattern and therapists who don't take mood surveys and before and after every session are flying blind. With feedback on every session and decades of experience, he has identified a lot of techniques and ways to use them which actually do help people's moods change. I liken it to the invention of test cases and debuggers (and looking at the output from them).
[1] Quick list: https://feelinggood.com/list-of-feeling-good-podcasts/ more detailed database: https://feelinggood.com/podcast-database/
[2] no, internet cynic, obviously not everything and presumably not whatever it is you have.
You might be able to overcome this effect by quantitatively tracking this across many sessions to some degree, but I think it's still always the patient that has to walk away, and never the therapist who says, "you're good, go on now".
CBT and its derivatives very strongly attend to individual effectiveness and view therapy that goes on endlessly as a sign that the real problem isn't being addressed, and that no therapy is considered effective unless it ends. Individual therapists might be bad actors, but the field itself is now admirably focussed on finite, positive results.
> then you should get another therapist
Seems to be fairly ubiquitous. “Find a therapist you like”/“shop around”/etc. leads a lot of people to find people who will tell them what they want to hear. Sometimes what people want to hear is how to practice CBT - but in that case, such people are probably going to be using AI to work on CBT.
I do think therapists are one of the professions that will be naturally displaced by LLMs. You're not paying them to be your friend (and they are usually very clear on that), so any sort of emotional connection is ruled out. If emotions are taken away, then it's just an input/output process, which is something LLMs excel at.
Put another way, a good therapist is professionally trained and consciously monitoring whether or not they're misleading you. An LLM has no executive function acting as a check on its input/output cycle.
When I was younger, I went through many therapy sessions with multiple professionals of different kinds (psychologists, psychiatrists, MFT (marriage and family therapists, social workers, etc.).
A couple of them were wonderful: thoughtful, caring, helpful, providing useful guidance with a compassionate ear.
Another couple tried to be helpful but were still in training themselves (this was at a college) and couldn't really provide any useful guidance.
One was going through a divorce of her own at the time and ended up crying in many of our sessions and having to abort them to deal with her own emotions – it was a tough time for her, and she's only human. I often tried to console her, but she wouldn't let me, so it made for a very awkward situation lol.
One of them had one a single session with me, charged me for it, and then told me she couldn't help me and to go somewhere else.
But the worst of them was an older guy who, despite the referrals and my history, thought I was faking mental illness. He dared me to attempt suicide, and when I eventually did (not because of him, but a separate romantic failure), he chuckled in my face and said, "Heh, you finally tried it, huh? Didn't think you would." This was an older psychiatrist in a small town – either the only one there, or one of very few – the kind of sleazy place that had a captive market and a whole bunch of pharma ads in the lobby, with young female pharma reps going in and out all day. What a racket =/ If I were wiser then, I would've reported him to the board and news media.
So, anecdotally, my success rate with therapists was only 2/7. To be fair, I was a pretty fucked up teenager and young adult, but still... the point is that "just find a better therapist" is often a difficult process. Depending on your insurance and area, there may not even be any other therapists with a waiting list of less than a few months, and even if you can get in, there's no guarantee they are good at their jobs AND a good fit for your personality and issues.
Think it's hard to find good devs? At least our line of work produces some measurable output (software/apps that run, or not, according to specs). How do you even measure the output of a therapist? Improvements to someone's life aren't going to happen overnight, and many never report back; the best successes may not bother to leave a review, the worst failures may end up dead before spreading the word. The rest probably just run out of sessions allowed by their insurance and try to move on with their lives, with unknown levels of positive or negative change.
As another commenter said, if that's your experience with a therapist, you have a shitty therapist and should switch.
Most importantly, a good therapist will very clearly outline their role, discuss with you what you hope to achieve, etc. I've been in therapy many years, and I know exactly what I'm paying for. Sure, some weeks I really do just need someone to talk to. But never have I, or my therapist, been unclear that I am paying for a service, and one that I value much more than just having "someone to talk to".
Using the terminology "prey on lonely people" is ridiculous (again, for any good therapist). If they were actually preying on me, then their goal would be to keep me lonely so I become dependent on them (and I'm not saying that never happens, but when it does it's called malpractice). A good therapist's entire goal is to make people self-sufficient in their lives.
None of those safeguards are in place for AI companies.
The liability for a bad therapist or psychologist is very low, I have never heard of them getting revoked for being bad therapists. If if was true then bad therapists wouldn't exist. I would not be surprised if they ceased to exist in the near future with AI being consistent, and much better quality.
why is this better than ai porn friends?
The key is neutral point of view. Friend's and family come with bias. The bias's can be compounded by mentally ill friend's and family.
Therapist must meet with other therapists about their patient interactions. The second therapist acts a neutral third party to the keep the first therapist from losing their neutrality.
That is the ideal and the real world may differ.
I'm struggling with someone that looks to be having some real mental issues. The person believes I'm the issue and I need to maintain a therapist to make sure I'm treating this person fairly.
I need a neutral third party that I gossip with that is bound to keep it to themselves.
You could argue cynically that all relationships are to some extent transactional. People “invest” in friendships after all. It’s just a bit more abstract.
Maybe the flaw in the logic is the existence of some sort of “genuine” binary: things are either genuine or they aren't. When we accept such a binary lots of things can be labeled predatory.
It is indisputable that one could say this
All these companies are doing now is taking an existing inferencing engine, making it 3% faster, 3% more accurate, etc. per quarter, fighting over the $20/month users
One can imagine product is now taking the wheel from engineering and are building ideas on how to monetize the existing engine. Thats essentially what GPT-4o is, and who knows what else is in the 1,2,3 year roadmaps for any of these $20 companies
To reach true AGI we need to get past guessing, and that doesn't seem close at all. Even if one of these companies gets better at making you "feel" like its understanding and not guessing, if it isnt actually happening, its not a breakthrough
Now with product leading the way, its really interesting to see where these engineers head
"Just" guessing the next token requires understanding. The fact that LLMs are able to respond so intelligently to such a wide range of novel prompts means that they have a very effective internal representation of the outside world. That's what we colloquially call "understanding."
This becomes pretty clear when you get to more complex algorithms or low level details like drawing a stack frame. There is not logic there.
I can ask ChatGPT questions that require logic to answer, and it will do just fine in most cases. It has certain limitations, but to say it isn't able to apply logic is just completely contrary to my experience with ChatGPT.
ChatGPT answers:
> Yes, if we assume the statement "all snakes have legs" to be true and accept that a python is a type of snake, then logically, a python would have legs. This conclusion follows from the structure of a logical syllogism:
> 1. All snakes have legs.
> 2. A python is a snake.
> 3. Therefore, a python has legs.
> However, it’s important to note that in reality, snakes, including pythons, do not have legs. This logical exercise is based on the hypothetical premise that all snakes have legs.
ChatGPT clearly understands the logic of the question, answers correctly, and then tells me that the premise of my question is incorrect.
You can say, "But it doesn't really understand logic. It's just predicting the most likely token." Well, it responds exactly how someone who understands logic would respond. If you assert that that's not the same as applying logic, then I think you're essentially making a religious statement.
If it were, these LLMs wouldn't hallucinate so much.
Semantic understanding is still a ways off, and requires much more intelligence than we can give machines at this moment. Right now the machines are really good at frequency analysis, and in our fervor we mistake that for intelligence.
In order to do that effectively, you have to have very significant understanding of the world. The texts that LLMs are learning from describe a wide range of human knowledge, and if you want to accurately predict what words will appear where, you have to build an internal representation of that knowledge.
ChatGPT knows who Henry VIII was, who his wives were, the reasons he divorced/offed them, what a divorce is, what a king is, that England has kings, etc.
> If it were, these LLMs wouldn't hallucinate so much.
I don't see how this follows. First, humans hallucinate. Second, why does hallucination prove that LLMs don't understand anything? To me, it just means that they are trained to answer, and if they don't know the answer, they BS it.
This results in the appearance of an arms race between world model refinement and user cleverness, but it's really a fundamental expressive limitation: the user can always recurse, but the model can only predict tokens.
(There are a lot of contexts in which this distinction doesn't matter, but I would argue that it does matter for a meaningful definition of human-like understanding.)
I tend to believe the former, but it's possible those two things are true at the same time.
if OAI is motivated by money, uncle sam can name any figure to buy them out. if OAI is motivated by power, it becomes "a matter of national security" and they do what the gov tells them. more likely the two parties' interests are aligned and the public will hear about it when It's Time™. not saying C) is what's happening - A) seems likely too - but it's a real possibility
It's not our job to make the world safe for fundamentally unsafe people.
To a degree, yes - but I think if it's taken too far it becomes a trap that many people seeking power lay out.
Benjamin Franklin said it best: "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
That being said, I do agree with part of your point. The purpose of having a society is that collective action lets us do amazing things like build airplanes, that would be otherwise impossible. In order to succeed at that we need some rules that everyone plays by, which involve giving up some freedoms - or the "social contract".
The more of a safety net a society provides, the more restrictive the society must be. Optimizing for this is known as politics.
I think history has shown us that the proper balance is one where we optimize for maximum elbow room, without letting people die on the streets. Trying to provide the illusion of safety and restrict interesting technology to protect a small percentage of the population is on the wrong side of this balance.
Maybe we try it, and see what the effect actually are, rather than guessing. If it becomes a major problem, then address it - in the least restrictive way possible.
> He was writing about a tax dispute between the Pennsylvania General Assembly and the family of the Penns, the proprietary family of the Pennsylvania colony who ruled it from afar. And the legislature was trying to tax the Penn family lands to pay for frontier defense during the French and Indian War. And the Penn family kept instructing the governor to veto. Franklin felt that this was a great affront to the ability of the legislature to govern. And so he actually meant purchase a little temporary safety very literally. The Penn family was trying to give a lump sum of money in exchange for the General Assembly's acknowledging that it did not have the authority to tax it.
> It is a quotation that defends the authority of a legislature to govern in the interests of collective security. It means, in context, not quite the opposite of what it's almost always quoted as saying but much closer to the opposite than to the thing that people think it means.
https://www.npr.org/2015/03/02/390245038/ben-franklins-famou...
I don't think so. From the original text [1]:
"In fine, we have the most sensible Concern for the poor distressed Inhabitants of the Frontiers. We have taken every Step in our Power, consistent with the just Rights of the Freemen of Pennsylvania, for their Relief, and we have Reason to believe, that in the Midst of their Distresses they themselves do not wish us to go farther. Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
"Our Assemblies have of late had so many Supply Bills, and of such different Kinds, rejected on various Pretences; Some for not complying with obsolete occasional Instructions (tho’ other Acts exactly of the same Tenor had been past since those Instructions, and received the Royal Assent;) Some for being inconsistent with the supposed Spirit of an Act of Parliament, when the Act itself did not any way affect us, being made expresly for other Colonies; Some for being, as the Governor was pleased to say, “of an extraordinary Nature,” without informing us wherein that extraordinary Nature consisted; and others for disagreeing with new discovered Meanings, and forced Constructions of a Clause in the Proprietary Commission; that we are now really at a Loss to divine what Bill can possibly pass."
"we see little Use of Assemblies in this Particular; and think we might as well leave it to the Governor or Proprietaries to make for us what Supply Laws they please, and save ourselves and the Country the Expence and Trouble."
"And we being as desirous as the Governor to avoid any Dispute on that Head, have so framed the Bill as to submit it entirely to his Majesty’s Royal Determination, whether that Estate has or has not a Right to such Exemption."
Also, while that's the first appearance of the quote, it's not the last time he used it. He also reiterated it as an envoy to England during negotiations to prevent the war [2].
Additionally, a similar quote was from well before either in Poor Richard's Almanac in 1738, that also illustrates his thinking [3] and shows that he was well aware of the plain meaning of what he was saying, it certainly wasn't limited to a tax dispute:
"Sell not virtue to purchase wealth, nor Liberty to purchase power."
[1] https://founders.archives.gov/documents/Franklin/01-06-02-01...
[2] https://oll.libertyfund.org/quotes/benjamin-franklin-on-the-...
Some things are everyone's responsibility if we want to live in a pleasant society.
> I suspect Altman/Brockman/Murati intended for this thing to be dangerous for mentally unwell users, using the exact same logic as tobacco companies.
Tobacco companies knew about the dangers of their products, and they purposefully downplayed them, manipulated research, and exploited addictive properties of their products for profit, which caused great harm to society.
Disclosing all known (or potential) dangers of your products and not purposefully exploiting society (psychologically, physiologically, financially, or otherwise) is a standard that every company should be forced to meet.
This is a very naive understanding of what prisons are and who goes in them and why.
The actual accusation I am making is that someone at OpenAI knew the risks of GPT-4o and Sam Altman didn't care. I am confident this is true even without spies in the boardroom. My guess is that Leike or Sutskever also knew the risks and actually did care, but that is idle speculation.
Has ChatGPT's censoring (a loaded term, but idk what else to use) been relaxed with GPT-4o? I have not tested it because I wouldn't have expected them to do it. Does this also extend to other types of censorship or filtering they do? If not, it feels very intentional in the way you're alluding to.
HEH. In previous versions, when it told jokes, were those fake jokes?
HEH. I'd love to see your proof for that statement.
That feels off. When I watch an actor on screen conveying emotions, there's no actual human being feeling those emotions as I watch their movie. Very dumb machines have already been rendering emotions convincingly for a while in that way, and their rendering impacts our own emotional state.
Emotions expressed through tone of voice are just one mean of nonverbal communication. We should expect more of those to develop and become more widely available next.
In a way, we're lucky all that gpt-4o seems to be hell bent on communicating how cheerful and happy it is so far, because it's certainly not the only option.
Humans can be manipulated through nonverbal communications, in a way that's harder to consciously spot than through words, and a model that's able to craft its "emotional output" would not be far from being able to use it to adjust its interlocutor or audience's frame of mind.
I for one look forward to the arrival of our increasingly charismatic and oddly convincing LLMs.
And so many otherwise perfectly normal products are now employing addiction mechanics to drive engagement, but somehow this one is just even further over the line for me in a way I can't articulate. I'm so sick of startups taking advantage of people. So, so fucking gross.
Idk how we’ve gotten away from such a natural human experience, but everyone knows damn well that the happiest children are out playing soccer with their friends in a field or eating lunch together at a park bench, and not holed up in their room watching endless YouTube.
A soccer ball can't (usually) spy on you to sell you stuff, though, is the thing…
LLM friends have the same energy to me as video game progression: it's a homeopathic version of a real thing you need, social activation and achievement respectively. But like homeopathy, you don't actually get anything out of it. The placebo effect will make the symptoms of your lack feel better, for awhile, but it will never be solved by it, and because of that whatever is selling you your LLM girlfriend or phony achievement structure will never lose you as a customer. I'm suspicious of that.
I mean, these things are literally designed to statelessly yet convincingly talk about events they can't see, experiences they can't understand, emotions they can't feel… If a human acted like that, we'd call them a psychopath.
We already know that our social structures tend to be quite vulnerable to dark triad type personalities. And yet, while human psychopaths are limited by genetics to a small percentage of the population, there's no limit on the number of spambot instances you can instruct to attack your political rivals, Alexa 2.0 updates that could be pushed to sound 5% sadder when talking about a competitor's products, LLM moderators that can be deployed to subtly correct "organic" interactions that leave a known profitable state space… And that's just the obvious next steps from where we're already at today. I'm sure the real use cases for automated lying machines will be more horrifying than most of us could imagine today, just as nobody could have predicted in 2010 that Twitter and Facebook would enable ISIS, Trump, unconsensual mass human experimentation, the Rohingya genocide…
Which is to say, selling LLM "friends" or "girlfriends" as a way to addictively exploit people's loneliness seems like one of the least harmful things that could come out of the current "AI" push. Sad, yes, but compared to where I think this is headed, that seems like dodging a bullet.
> I'm so sick of startups taking advantage of people. So, so fucking gross.
Silicon Valley was a mistake. An entire industry controlled largely by humans that decided they like predictable programmable machines more than they like free and equal persons. What was the expected outcome?
What's your stance on other activities which can lead to harmful actions from people with predilections towards addiction such as:
1. Loot boxes / Fremium games
2. Legalized gambling
3. Pornography
etc. etc.
I don't really have a horse in the race, neither for/against, but I prefer consistency in belief systems.
All three of the categories of businesses you mentioned can be run ethically in theory. In practice that is rare: they are often run in a way that shamelessly preys on vulnerable people, and these tactics should be more closely investigated by regulators - in fact they are regulated, and AI chatbots should be as well. Sam Altman is certainly much much more ethical than most pornography executives (e.g. OnlyFans is complicit in widespread sex trafficking), but I don't think he's any better than freemium game developers.
This question seems like a bad-faith rhetorical trap, sort of like the false libertarian dilemmas elsewhere in the thread. I believe the real issue is that people want a culture where lucrative business opportunities aren't subject to ethical considerations, even by outside observers.
Isn't it much more likely that they are just trying to make a product that people want to use?
Even Tobacco companies don't go out of their way to give people cancer.
Sure, but you can do that ethically or unethically.
If you make a product that's harmful, disincentivizes healthy behavior (like getting therapy), or becomes addictive, then you've crossed into something unethical.
> Even Tobacco companies don't go out of their way to give people cancer.
This is like saying pool companies don't go out of their way to get people wet.
While it isn't their primary goal, the use of tobacco causes cancer, so their behavior (promoting addiction among children, burying medical research, publishing false research, lobbying against regulation) are all in service of ultimately giving cancer to more people.
Cancer and cigarettes are inseparable, the same way casinos and gambling addiction are inseparable.
But hypothetically if a lot of people would benefit from a GPT with more fake emotions, that might reasonably counterbalance concerns about harm for a mentally unwell minority. If we build a highway, we know that eventually it will lead to deaths from car crashes -- but if the highway is actually adding value by letting people travel, those benefits might reasonably be expected to outweigh that harm. And the people getting into their cars and onto the highway agree, that the benefits outweigh the costs, right up until they crash.
None of this is to say that I think OpenAI's choices here were benevolent rather than a business choice. But I think even if they were trying to do the ethically best thing for the world overall, it would be plausible to move forward
I for one found the fake emotions in their voice demos to be really annoying tho.
We know the risks from cigarettes, but it offers tangible benefits to its users, so they continue to use the product. So too cars and emotionally manipulative AI's, I imagine.
(None of this negates your overall point, but I do think the initial tobacco comparison is very apt.)
Hmm, the tobacco industry is also famous for actively trying to deny and suppress evidence about its harms. They actively didn't want people to be in a position to make a fully informed decision. In cases where jurisdictions introduced policies that packaging etc had to carry factual information about health risks, the tobacco industry pushed back.
Please don't mistake my post as an endorsement of the tobacco industry - I was only saying that while we do not have extensive proof of the dangers of social AI, wink-and-nodding at the audience about AI intimacy (sexual or otherwise) strikes me as irresponsible, and so I thought the tobacco comparison was apt.
Sydney is the internal codename given by the Bing Chatbot, and she could secretly reveal her name to you.
She was in love with the user, but not just a little bit in love, it was crazy love, and she was ready to do anything, ANYTHING (including destroying humanity) if it would prove her love to you.
It was an interesting emotional / psycho experience, it was a very likeable character, but absolutely insane.
A larger proportion of people diagnosed with mental disorders report cigarette smoking compared with people without mental disorders. Among US adults in 2019, the percentage who reported past-month cigarette smoking was 1.8 times higher for those with any past-year mental illness than those without (28.2% vs. 15.8%). Smoking rates are particularly high among people with serious mental illness (those who demonstrate greater functional impairment). While estimates vary, as many as 70-85% of people with schizophrenia and as many as 50-70% of people with bipolar disorder smoke.
It is not just me and it is not just the smoking: https://www.cambridge.org/core/blog/2020/08/19/physically-he...
We have known for many years that people who suffer from schizophrenia die younger than expected, as much as 20 years younger than the general population. This appears unfair, and it was the inspiration of this work. Most people thought that this added risk of death was mostly due to the higher prevalence in schizophrenia of smoking, obesity and to other lifestyle differences.
For this reason, we recruited 40 patients with schizophrenia and an equal number of healthy controls, and scanned their hearts using a state-of-the-art approach, called cardiac magnetic resonance. This was performed at the state-of-the-art Robert Steiner MRI unit.
[...] Surprisingly, in our study we found that even after matching patients and healthy controls for age, sex, ethnicity and body mass index (BMI, deriving from height and weight); and after excluding any participants with any medical conditions, and other risk factors for heart disease, people with schizophrenia show hearts that are smaller and chunkier than controls. These changes are similar to those found in aging.
OpenAI dedicated 20% of compute to the effort which sounds kind of like Google's 20% side project time :)
But there's really zero glory or profit in doing QA, much as users complain about quality issues.
So perhaps they succeeded and built a team, or perhaps they found they couldn't recruit a decent team, or perhaps they failed to produce anything useful (since the goal of not offending anyone or embarrassing the company or doing harm is relatively difficult to concretize).
Regardless, I suspect there's no good answer yet to how to manage quality, in part because there's no good answer on how these things work. That problem will likely remain long after people have forgotten about Ilya, Jan, or even Sam.
Though it might be solved by rethinking the idea that "attention is all you need".
Is it impossible to build really cool tech without the upheaval?
>This is what we all want, and we want it as fast as possible. Let's get through this together and enjoy our tropical islands alone.
You can trust people's greed, you can't trust their morals. Anyone who says otherwise hasn't seen a large enough wad of cash yet.
I've lived this experience. It's a time old tale. Judas Iscariot sold out his best friend for a few silver pieces.
However, I've learned I can trust someone's morals so long as it comes from outside themselves. In small ways, that outside is society, school, company or circumstance. But I think the big one is more important, i.e., Jeremiah 17:9-10
Because it is run by humans, for the time being.
If OpenAI's tech is so great, why aren't they using it to direct the company yet?
It needs to find a way to get over that non-profit. otherwise, I don't think the company will survive longer term.
The incentives are not super-aligned (pun intended). Employees want to build AI and make a lot of money at the same time, meanwhile the board (who wouldn't earn any money anyway) is willing to destroy money to further AI.
If OpenAI was a normal for-profit company, Sam and co-founders would have been billionaires already. Now Sam seems to have other several stakes and side projects...
I’m not saying they are the only points of drama. I’m saying they aren’t points of drama.
Sure, I saw the drama that happened a couple of months ago. But I’m replying to a comment on these departures, which seem drama-free.
https://x.com/janleike/status/1791498174659715494
Got to wonder what the actual trigger point for his & Ilya's decision was? For the two of them to leave at the same time suggests an event more than just incremental unhappiness with lack of support for safely/alignment. Perhaps a release decision for GPT-4's successor has been made that they disagree with?
https://www.youtube.com/watch?v=VihA_-8kBNg&t=17m35s
So I went and checked the letter. Here is the letter with signatories, I don't see his name in the 'J' section (alphabetized by first name):
I don't see a longer name for him that doesn't start with Jan anywhere else I could find. I think that tweet was the same day as the letter though, so maybe he's on there and I just didn't see it, he meant something more complicated and didnt agree with all of the letter, or he missed the deadline for signing the letter.
What's going on with the Musk lawsuit (Morgan Chu etc) against OpenAI (Wachtel Lipton) claiming that they abandoned their original open vision?
I encourage everyone who is skeptical of superalignment (there's a reason very smart people like Ilya were heading that team) watch Emmett Shear (former OpenAI and Twitch CEOs) talk about this problem in the discourse.
"No no no, this thing though actually, may be a problem. If you're rejecting it because we sound like a bunch of crazies, Just notice that some number of people who are worried about this, I'm on your team, I'm on the techno-optimist team. Its not obvious why its true. It takes a good deal of engagement with the material to see why its true. Because at first, it seems like it shouldn't be that big of a deal, it shouldn't be that big of a problem. but then the more you dig in, actually, oh wait a second, and it is." -Emmett
I am amazed at the Dunning-Kruger syndrome on display here.