They stole my voice with AI
jeffgeerling.com
jeffgeerling.com
My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob.
And this was before AI was easy to access. You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread. You can't just give a toddler a knife and then blame them for stabbing someone.
Has nothing to do with fame, with security, with copyright. This will get people killed. And we have no tools to control this.
https://x.com/search?q=blasphemy
I fear the future.
The same way it took social media like reddit a few years of "finding the culprit" / "name and shame" till mods figured out that many times the online mob gets it wrong and so now that is usually not allowed.
But many people will suffer this until laws get passed or it enters into common consciousness that a video is more likely to be fake than it is to be real. Might be more than 5 years though. And unfortunately laws usually only get passed after there's proven damage to some people from it.
This kills the medium.
Just as ubiquitous scam calls have moved people away from phones, this moves people away from using media which cannot be trusted. Done enough this destroys reporting and therefore democracy. I wonder when the first nonexistent candidate will be elected.
Check the twitter link, you won't have to scroll much to find a mullah being blasted for blasphemy. No one is safe.
Mobs are not formal organisations with leadership and membership dues.
Mobs are general everyday people who are sick and tird of this miserable life and need an excuse to rabble and rouse.
So in this case, fake dead person's blasphemy will only be channeled towards their family members, mobs just need an excuse to get crazy.
AI is not what causes mobs, it just helps lower the incitement barrier by providing quality bait.
Make them into fools.
- Stone the blasphemer!
- Who? The one that died las year? The one that died in the war in '94? The one that left 4 years ago? Don't you have anything better to do other than fooling yourself and spreading more rumors?
Doesn't make it any less dangerous while the mass is being vaccinated against bullshitters.
More alternatives: fake full persons. The mobs will be confused when they are directed to hunt down someone with a family name in a neighbor where no one has that family name. Furthermore, send them to the car repair shop between the coffee shop and the market in a street with no car shops, markers and coffee shops.
> what if AI was used to imitate a person saying something blasphemeous?
Of course, this could be misused to post something with plausible deniability, but if you want to say something controversial, why wouldn't you make another account for that anyway?
I know that one could theoretically sign posts with GPG, but it would be much nicer and less noisy if sites would have UI to show something like: Signed by <fingerprint>, key used for N years.
One issues is that most social media want your identity to be the account on their service and not some identity (i.e. key) that you control.
> There should be a way to cryptographically sign posts (everywhere).
Key signing will not be the solution because it isn't addressing the problem. It exacerbates it.
Ah, think I see your point. Your worry is that language use, etc. could be use to deanonymize you, by correlating with text that was not written anonymously. But that's a separate issue from voice or writing style cloning to pretend it's you that said it. In the latter case you could use a pseudonymous signing key?
I agree that deanonymization is an issue that is hard to tackle. I wonder if someone studied how unique writing style is. E.g. browser fingerprints are fairly unique, but I wonder to what extend you can filter a person from, say a pool of 100 million, using writing style alone (grammar, vocabulary use, etc.). I guess it becomes quite easy if you engage in a lot of domain-specific discussions and use their vocabularies.
E.g. if I'd talk about Marlin-kernels here, you could probably narrow me down to a few hundred people. Throw in another comment about the Glove80. Maybe ten people at most?
> I wonder if someone studied how unique writing style is.
Your language is in fact a fingerprint. And like you suggest, topics too. Much of our freedom of anonymity comes from the fact that it is hard or not worth it to dox people.
I do agree that verification is a different issue though. I'm not sure keys will solve it because you're not going to sign anything that is scandalous, so it might even give evidence for those that want to falsely claim foul play. And how do you sign a leak?
The problem with signing is that it seems to work for the cases we don't care about and do nothing for the ones we do. That is unless we sign literally everything, including our voice, but then you kill anonymity (why I connected it) and you could then probably clone that too.
If stylometric analysis runs on all comments on the internet then yeah.
Bad things will happen, very very bad.
I honestly think it should be at least illegal to do this kind of analysis because it'll be a treasure trove for the commercial sector to mine this data correlated to real people not to think of the destruction in millions of people with personal anonymous blogs etc.
Actually thinking about it further you could also easily group people political affiliations, and all kinds of other thoughts, dark, dark stuff!
Meanwhile gangsters or nation states are probably already working on automated deanonymization and blackmail at scale, but will target the dark web before the public one. Not sure any of this stuff is that changed by the advent of deep fakes and llms tho, probably just boring classical statistics does ok?
But I'm pretty sure this would make life very bland. Slow down innovation as we become less creative, less willing to deviate from the norm, and the Overton window of what's socially acceptable narrows. After all, we do love having enemies in some way or another, even if it means making them up.
So far I get a lot of pushback and statements like "it's just linear algebra" and "quality doesn't matter, it's quantity. Just scale. That's the bitter lesson". The former tells me they know very little linear algebra and the latter tells me they know very little about probability distribution and the concept of coverage... I routinely see papers get awards and tens of thousands of citations when they're making errors I would expect undergrads in my classes to do better at... I think a big reason I'm frustrated is that supposedly we share the same goals -- building AGI -- but I'm told I'm stopping them from doing so while they reject my papers and theirs gets accepted... (Fwiw I'm happy to accept scale papers. There should be a wide range of ideas explored. But I'm upset that we're on a railroad and any deviation from that is considered heresy)
Already happening at Google and Meta for a decade or more.
Now imagine that account was linked to a SIM. It's trivial for a nefarious actor to get it re-activated, infact there was a video by Veritasium just today where they didn't even need your SIM.
But even if they are not that hi-tech, it's not that hard to get a SIM issued in your name, or other hacks of a similar nature, we have all heard of stories.
Worse, you lost that SIM a decade back, the number gets back into the queue, and is eventually re-issued to someone new... and they try to create a facebook account, and are presented with yours.
They can then re-activate your old facebook account, and post a video/audio/text of "godelski" saying they like pineapple on pizza. and before you can defend yourself, the pizzarias have lynched you.
(I dare not use a real example even as a jest, I live here)
Are you 100% sure of all your old social media accounts, all the SIM you have ever used to log-in to accounts?
We leave a long trail.
While it is impossible to take back a disclosed secret, we can create a legal framework which issues immediate business-stopping corporate death sentences for spreading that data without consent (or after you revoke consent, yes it needs to be revokable). Your data is so valuable, because now that we have universal function approximators you can be simulated (to a prediction horizon) and that simulation interrogated.
Not a great outlook though, if everybody does this...
Besides, topics and shared details are also pii. It's more that they aren't useful in that way until you have scale.
Plus, on HN you can't edit posts after some time.
at this stage and current behavior, I wouldn't mind some poison. That's the only way companies will learn at this point.
I suppose even a throwaway could be linked to my identity if a comment was long enough, but probably only with some limited certainty.
We're human, we all have stuff to "hide". I might want to vent about my boss or work and if they heard it that would certainly take it different than it was intended. We even complain about our friends, especially our close friends. Because no relationship is without conflict. But being human we often have to actualize our feelings with words, because we're social creatures. I'll at times complain about my partner, but that doesn't mean they also aren't my favorite person in the world.
To be human requires some anonymity. And no one should feel like they're going to be scrutinized for every little thing they say.
Yes this violates any EU citizen's right to be forgotten under GDPR. Welcome to silicon valley.
Sort of moot point considering the multiple HN archives that would still have the original username attached
Under which legal theory does EU law apply here?
I could be hallucinating, but I swore job boards also had peopel hire from the EU.
> You can't nuke your account. You can close it but your comments remain on the site.
I'm sorry, but this is a cope out. The "lynching from apparent cultural deviation" is something that needs to be moved on from. Developed countries do the same too to some extent, with "cancel culture" and such.
There are ways to have progress in this, and, well, to feed someone's entrepreneurial spirit, it's one of those really hard problems that a lot of people, let's say, "a growing niche market", needs it to be solved.
But Blasphemy by whatever means, is one of the tools by which society sets certain boundries, and it's really hard to move away from a model that worked so 'well' for us since the first civiliations.
> My country is already has blasphemy lynching mobs
in your case the problem is not AI, it’s your country.
It’s like if someone said “I’m scared of someone bringing a semi-automatic weapon to my school and doing a mass shooting. My country has lax laws about guns and their proper use”. And then you said “in your case the problem is not guns, it’s your country”.
I mean, it’s technically true, but also unhelpful. Such ingrained laws are hard to change and you can be placed in danger for even trying.
Before someone decries the gun example as not being comparable, it is possible to live in a country with a monumental number of guns and not have mass murdering every day. It’s called Switzerland.
But let’s please stick to the subject of AI, which is what the thread is about. The gun example is the first analogy which came to mind, and analogies are never perfect, so it’s unproductive to nitpick the example. I don’t mean to shift the conversation from one contentious topic to another.
Vs the US, where there are loopholes that you can use to avoid even a basic background check, and then use it for the very first time to shoot someone.
Second, you learn a certain respect for the firearm and are expected to observe strict safety rules when handling it. That gives you a kind of psychological flinch when you consider doing anything out of the norm with it.
The difference might be in what guns mean to Swiss people vs. the US.
A gun in the US has this "if the government becomes destructive of these ends we can start shooting" connotation. A gun is there for self defense. Someone doesn't get off your law, you use your gun.
In Switzerland the gun and the compulsory military service you mention is there for the people to protect their country and fellow countrymen. You are trained in defending your neighbor, who just stepped on your lawn against outside aggressors.
The US has both problems: widespread availability of weapons and a high level of freedom to incite violence through spreading lies about groups. Which is why it sees much more of these incidents than countries which have similar levels of gun ownership.
The non-gun version of the problem is mass stabbings, which are less lethal.
If an AI fake-porn of some ordinary person involving a minor was unleashed, think of the utter shame and horror they would be treated by people for the rest of their lives, even if it were proven false.
No one would believe them, work with them, hire them, rent them, they would wish they had been lynched instead of the life they live.
It happens once, then some wag creates videos of all the mob leaders blaspheming or whatever. Undermines the idea that the videos are the root cause.
Also there are no mob 'leaders'. These are everyday folks, these are you and me and my neighbours.
A student pissed at a teacher could create a fake AI video of his teacher at night, and wake up to no pesky teacher and feel no remorse over what happened while he slept.
The student was not a mob leader, just some dumbass who was angry at their teacher for giving bad grades, and had access to simple AI tools (soon available online for a trivial fee, if not free).
____
Be honest, are you 100% sure of the status of that facebook account you deleted in 2018? Zuck never deleted it, that content is still there to be mined and abused. Also not that hard to just log in to the damn thing and post something spicy.
I don't remember if old posts can be edited in FB, cause that would allow the person to really gild the lily.
I believe you are right. When the first unbelievable clips appeared on Facebook (where the most gullible people are), you could see the old ladies cheering "oh how wonderful! where did you film that?" Now in the comments section you will mostly see people saying "sheesh, stop polluting my feed with these fake AI-generated spam". Don't get me wrong, many people still fall for them, but the tendency seems clear - after all, we managed to survive because of learning.
Remember the post is about cloning a youtuber's voice using AI; most people were thinking of the copywrite aspect of it, but I wanted to share with people what my fear was.
My fears are, things are already bad w.r.t lynch mobs before AI, after AI things will only get worse with fake but realistic sounding voice notes etc. That's my fear, doesn't have to be yours.
You don't have lynch mobs (hopefully) but surely you can think of other problems that a fake AI image/audio/video can cause, and just low the barrier is, and just how good the tools are.
It’s fundamentally a societal issue, not a technological one. Yes, it’s scary what effect AI could have on lynch mobs, because lynch mobs are scary.
The only way forward is making sure most people know that digital content can be easily faked. If you hand out good faker tools for free, it will happen faster
Remember when people believed that camera took our soul when we are caotured on image?
Or https://www.npr.org/2024/09/19/nx-s1-5114047/springfield-ohi... , where repeating racial libel causes a public safety problem.
While this kind of incitement in no way requires AI, it's certainly something that's easier to do when you can fake evidence. See also https://www.bbc.co.uk/news/articles/c5y87l6rx5wo
In the US, freedom of speech means the government won't stifle you, but the court of public opinion will crucify you for saying the wrong thing. In the linked case, a school principal was framed for being a racist, a fire able offense. Better than being killed by religious extremists, I suppose, but still not great. Thankfully in this case they were able to find the perpetrator who faked evidence, but we have a problem.
The solution is not to ban technology (it's impossible), but to create a more decentralised society where the majority of voters, which is easily brainwashed, doesn't get to dictate the life of the minority. We need to destroy and dismantle all the gigantic entities that ruin our life for the advantage of the few, whether they are the government or corporations ruled by billionaires controlling the government.
We also need to create a better society with less mental illness, where people are not depressed by how unfair life is that they go on and kill kids in school - but I believe if we reduce large societal inequality this will sort itself out over time.
Parents in my circle are already banning social medias for their kids until they're adults - so over time we'll evolve to get better at ignoring all the crap we get exposed to.
That take pretty much throws the shooters' preceding years of anguish under the bus to make a political score. "Ban the wrongspeech!" is a seductively easier proposition than ending youth bullying and ostracism.
> youth bullying and ostracism
... which itself counts as free speech?
A lot of people get bullied in a lot of countries. Few of them turn to mass murder. The mass shooter phenomenon is mostly America, plus a few people who've been radicalized by the same racist discourse on the same message boards (Anders Brevik, NZ mosque shooter).
In quite a lot of cases closer investigation suggests that they were ostracized precisely because they'd already chosen hate.
So yeah, I would call this "wrongspeech" and wrongspeech is already banned.
Sure, distrust everything digital, but what if only evidence of someone doing something wrong is digital?
With celeberaties, atleast they are famous enough that the idea of fakes exist... but you are Johnny nobody, that's not a fake, you just got caught and don't want to admit you said/did it!
The court of public opinion is much smaller and personal, and 'evidence' is much more realistic and detailed.
Nothing new.
The trick is being rich without being famous!
In the UK a government was just elected with a historical absolute majority by only ten million people, and now first time offenders are being sent to prison for making stupid offensive statements online.
> In a now deleted post on her X account, Mrs Connolly, from Northampton, wrote: “Mass deportation now, set fire to all the f*** hotels full of the bastards for all I care… If that makes me racist, so be it.
> The court heard Kay copied, pasted and uploaded Mrs Connolly’s post at 12.27pm on Wednesday from a BBC News report and added the hashtags #standwithlucyconnolly #lucyconnolly #f**northamptonshirepolice #conservative #FaragesRiots #RiotsUK and #Northampton.
> As well as a post which urged people to “mask up” during a protest targeting an immigration law firm, Kay tweeted to his 127 followers at 2.34am: “That’s 100% the plan, plus gloves. No car either so no number plate to trace and a change of clothes ready nearby.”
https://www.independent.co.uk/news/uk/crime/northampton-bbc-...
My point was that AI faked content could certainly get you locked up in the UK.
As far as I can tell the collective conscious of every country is swayed by propaganda.
A written headline is enough to incite rage in any country much less a voice or video indistinguishable from the real thing.
Folks in “developed’ countries have their lives destroyed or ended all the time based on rumors of something said or done.
Do you know "The boy who cried wolf"? Fabricate some allegations yourself and this will train people to disbelieve them.
You are assuming that people who are part of lynch mobs have the critical thinking skills to differentiate between real vs fake, and use logic.
Reminds me of the post I read on twitter, of some Thai/Chinese New Yorker whose mother told him not to speak Mandarin in public when COVID related Anti-Asian hate was rampant....
And he had to explain to her that she can't expect the sort of person who hits a random Asian to differentiate between Thai and Mandarin.
I could then trivially generate pictures or even videos of you e.g. by knowing your name. Of course that's just an example but I do think that's where we are headed and so the concept of "trust" will change a lot.
Once it's 2 clicks away to generate a believable video of someone at the kkk kitten barbecue getting along with ted bundy and jeff epstein, surely the evidence value of that would dwindle, and the brief period in history when video evidence was both accessible and somewhat believable will come to an end.
Being savvy means not treating photos/videos as "untrustworthy", which (ironically) is one of the things the Western world made fun of the Taliban for -- their fundamentalist view called it "haram":
https://www.deseret.com/1997/10/6/19338048/taliban-army-bans...
I worry about written matter too, and here as well, vast masses of LLM-spewed text will surely pollute the landscape irreversibly.
https://www.dallasnews.com/news/watchdog/2021/03/19/its-mind...
To explain for a more developed country context, the fakes that previously required skill in Photoshop and Audacity etc now is much simpler to implement with AI, allowing far more dipshits to create and share fake image/audio/video of someone they are pissed at during their lunch break on their phone.
That's way too quick, allowing people to shoot far too many arrows in a huff, before their reasonable brain has time to make them realise the consequences of their actions.
Tolerance of intolerance is a bullshit strategy that won't work.
Most (not all) religions are like a cancer, their main contribution to society is dividing people into US vs THEM.
If your religion encourages you to kill someone, MAYBE that’s a problem (not AI)?
I do sympathize that it is making it EASIER for them to do more harm. That’s a bit concerning to me.
I was thinking about deepfakes and how to protect from them and my conclusion is; there is no way you can protect from them from the practical point of view but from the legal point of view, governments can make laws where everything shared by somebody else that involves you is presumed fake unless there is substantial circumstantial evidence that says otherwise e.g. witnesses, legitimate metadata etc. etc.
Even before LLMs and GenAI, Photoshop became a synonym for messing around and faking photos so there is nothing new here but now there is more powerful "faking" software available to the masses.
Before computers and digital mass media sharing some compromising photo or tape could've been assumed authentic but now in the era of computers and software there is no way you can tell that something is authentic for sure.
>What if a third party gains access to your social media account(s) and starts posting fake content from there?
You can cause chaos and bad press in the short-term but when the original owner of the account restores ownership of the account everything falls apart. Like the commenter below said it happens all the time and it doesn't have any real impact on anything whatsoever.
That, and cryptographic materials being used to sign stuff too.
I think that's possibly the best we can hope for from a technical perspective as well as waiting for the legal system to catch up.
But the point was, doesn't have to be lynch mobs, for developed countries, even 'lesser' consequences are stull terrifying.
Imagine everyone in your office giving you the stink eye after lunch and you being fired by HR 'at-will', and not knowing what exactly caused it, after all the day started so normally...
Your account is a week old and half your comments are dead. I recommend referring to the guidelines.
Sure, not lynch mobs, but AI-generate fake media can certain ruin people's lives, and unlike photshop etc, the barriers of skill and time required are very low, and the quality is very high.
I share my country's experience because I wanted to share my personal perspective and fears, but please don't under estimate how AI can affect you. Just because you won't be death doesn't mean they can't turn you into a social pariah with a few clicks.
The religion isn't the (whole) issue here, this situation can apply in the secular West just as easily. The punishment won't be death, but it can still ruin people's lives. A fake pedophilia accusation comes to mind, where even if proven innocent you'll still be royally fucked for the rest of your life unless you spend considerable expense and effort.
Of course.
I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something.
I think it might be a rough couple years but hopefully we'll be through it soon.
Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?
>During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice.
>Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice
https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7...
Worse, there isn't an attitude of default skepticism in many areas/cultures. If a person is suspected of violating the moral code the priority will be punishment and reinforcing that such behavior isn't acceptable. Whether or not the specific person actually did the specific act is a secondary concern.
It's just going to increase the number of people who will be harmed or killed.
In a lawyers view, and a judge's view, some skilled expert "hackers" can, and its called hacking. (so i guess we're all hackers)
I once discussed these things with a (knowledgeable) lawyer. He explained you can just present almost anhthing in a court case, and when it isn't refuted, well then it's valid.
In a case my lawyer (same one) presented a printed out email. Other party did not claim it was false, so it's suddenly just as valid as a registered letter. (it was a genuine email).
In another unrelated case, the other party suddenly introduced a forged picture. If I hadn't been there at that moment (I wasn't supposed to actually), then suddenly it would have been proof.
Court cases are not about truth, and not about justice. They are about convincing the judge.
I'm currently the originator and lead on a project that will have huge impact on our bottom line. The team is mostly junior people, so when I presented the plan the response I got wasn't "I didn't think we could do that", it was "I didn't even know that that is a thing".
There used to be a joke about Rumsfeld's "known unknowns" vs. "unknown unknowns", but it's so true. And to the vast majority of people in the world the question isn't "how" to change a timestamp, it's understanding that a timestamp can even be changed.
I also however don’t believe that deepfaked porn is actually harmful. People are already imagining people naked, and it’s been easy to photoshop a nude for 15 years. A talented artist was needed 200 years ago. Allowing this (admittedly rude and disrespectful) act to be done with less skill changes very little.
Blasphemy laws—and the violence that sometimes accompanies them—are a cultural issue, not a technological one. When the risk of mob violence is in play, it's hard to have rational discussions about any kind of perceived offense, especially when it can be manipulated, even technologically, as you pointed out. The hypothetical of voice theft amplifies this: If a stolen voice were used to blaspheme, who would truly be responsible?
This is why we must resist the urge to give into culturally sanctioned violence or fear, regardless of religious justification. The truth doesn’t need to be violently defended; it stands by itself. If a system cannot tolerate dissent without devolving into chaos, then the problem lies within the system, not the dissent.
“An appeaser is one who feeds the crocodile, hoping it will eat him last.” - Winston Churchill
Sure we have mobs and you don't, but we are talking about AI here.
Infact let's imagine a totally different culture to illustrate my point.
Imagine you are an Israeli, and people in your office have a habit of sending Whatsapp voice notes to confirm various things instead of calls, because that way you can have a record but don't have to type every damn thing out. Totally innocent and routine behaviour, you are just doing what many other people do.
A colleague pissed at you for whatever damn stupid reason creates a fake of your voice saying you support Hamas by using said voice notes, using an online tool that doesn't cost much or require much... are you saying just because you won't be lynched, that there isn't a problem?
You are confused why everyone is pissed at you and why suddenly your boss fired you, and by the time you find out the truth... the lie has spread to enough people in your social circle that there is no clearing your name.
Think of how little data in voice samples is required to generate an audio clip thats sounds very realistic, and how better it will get in an year. You don't need fancy PC or tech knowledge for that, already websites exist that do for cheap.
Just because you weren't lynched is no solace.
People are the problem, AI is just providing quality tools with minimal skill and cost required, thus broadening the user base.
The problem of people manufacturing evidence by using synthesized voices should eventually result in audio voice recordings losing importance from an evidentiary standpoint. In fact, the quicker the use of it occurs, the quicker it will get devalued. And that is good. Someday soon, someone who sounds like the CEO won't be able to drain the company bank account based solely on trusting his voice. And hopefully, along the same lines, a voice recording of Person X blaspheming will lose its evidential power since it will be so easy to manufacture.
It's in my profile :)
Even if blasphemy is illegal in your country, people would probably agree that falsely accusing someone of blasphemy is also wrong.
In a more serious vein, this is definitely about unleashing an extremely powerful technology, at scale, for profit, and with insufficient safeguards (imagine if you could homebrew nuclear weapons -- that's inconceivable!)
There will be collateral damage. How much, and at what point will it trigger some legislation? Time will tell.
I recall photoshop blackmailing stories where usually woman were the target. Now literally "everyone" knows pictures can be manipulated/photoshopped. It will take a while yes, but eventually common folk will learn that these audios/videos can't be trusted.
> ...these observations hold true of singing, especially singing by a singer of renown. The singer manifests herself in the song. To impersonate her voice is to pirate her identity...
> We need not and do not go so far as to hold that every imitation of a voice to advertise merchandise is actionable. We hold only that when a distinctive voice of a professional singer is widely known and is deliberately imitated in order to sell a product, the sellers have appropriated what is not theirs...
They aren't going to use AI to have Tom Cruise in their film. He won't sign these rights away.
But they sure as hell want to have the next Tom Cruise sign those rights away as a condition for being hired to be Random Bystander #4 in a straight-to-streaming C-film.
Then, once he becomes successful and famous, they won't have to pay him a cent to keep using him, forever.
I can't wait for the future where even more of the wealth people who do work generate will be siphoned off to the owners.
Call your congressperson, ask them to co-sponsor and/or vote for it.
https://www.cbsnews.com/losangeles/news/california-bills-pro...
https://salazar.house.gov/media/press-releases/salazar-intro...
https://files.constantcontact.com/1849eea4801/695cfd71-1d24-...
Sometimes all you want is the effect, other times it's important that you're accurately representing effort or accounting for other human considerations.
Society is about compromises and balancing different needs against each other. Sometimes we go one way, other times we go the other way, there is no one principle that always solve any situation.
the current lobbying and legislating efforts seek to outlaw pens and pencils produced by anyone but a handful of US corporation, who only let you use their pens and pencils if you let them look over your shoulder while you write your squiggles.
So I’m increasingly of the opinion that it’s not the tool that needs to be regulated, but the use of the output.
Clone voices? Fine. Clone voices for deceptive or commercial purposes without the person’s consent? Not fine.
But then how do you prove it, what is deceptive, what is non-consenting voice cloning, yadda yadda.
I imagine we will shortly see a raft of YouTubers adding “do not clone my voice” notices to their channels like the Facebook “by posting this notice you remove all rights for Facebook to steal the copyright in your photos” spam posts that were doing the rounds at one point.
A lawsuit from a private individual or business entity is very different from the federal, state, or municipal government attempting to silence you, the latter is prohibited by the First Amendment.
I find it appalling that this needs to be spelled out.
Fully aware of the difference between a civil lawsuit vs government lawsuit though so you can pop your italics back in the box and rest easy!
Actually that's easy: it wouldn't be profitable, yet.
Anyway, given how the election is shaking out on Twitter, I have a feeling political usage will spark legislation and precedent far before commercial usage does. But that’s just a plain guess
If you got Bob who sounds awfully like Mr Very Famous Guy to record vocal that you then use to train your AI and use that vocal clone to sell your nutritional yeast extract as though it’s Mr Very Famous Guy selling it that would likely be a problem.
If you used the vocal clone to sell it, but said something like “oh hey it’s Bob here lots of people tell me how much I sound like Mr Very Famous Guy but I’m not him” then Mr Famous might have a case for his name being used without permission, but probably not the vocal clone.
But it’s all so new and there’s no precedent.
Given the lawyers are all busy working out whether using copyright protected books and music to train generative AI is legal or not - and have good arguments on both sides - it’s all a bit unclear how stuff like this will work out in the end.
The point is that Geerling is doing Youtube videos in the same field as the products this company is using his voice for. That makes it appear as if Geerling would be endorsing their products. If his voice had been used for, say, a nature documentary then people wouldn't have made the connection that it would have been him.
Sounding like someone certainly isn't illegal and a foreign company isn't going to respond to a lawsuit that aims to determine the source of their training data.
You'd just not have to pretend to be that person in a deceptive way (parody is legal).
Politician’s careers live and die in the fickle Court of Public Opinion. They’re probably the most susceptible cohort to AI fakes.
One of the rare times, it seems, that politician’s incentives are aligned with the populous. (Yes, I could have left that last part out.)
The Camry class needs its defenders, I wholeheartedly agree, but it’s also a core principle of contemporary praxis that you gotta let people choose their comfort level/ability to contribute. Encourage, promote, embolden — but try not to shame :)
Anyway, something tells me this blog post is gonna be more than enough. I don’t think basically anyone is on the side of stealing people’s voices, it’s just intuitively icky in a way that scraping the NYT and deviantart archives for training data isn’t. Public shaming isn’t gonna win him a big sack of damages, but it doesn’t seem like that’s what he’s after!
One can't help but wonder what theft even means any more, when it comes to digital information. With the (lack of) legal precedent, it feels like the wild wild west of intellectual property and copyright law.
Like, if even a superstar like Scarlett Johansson can only write a pained letter about OpenAI's hustle to mimic her "Her" persona, what can the comparatively garden-variety niche nerd do?
Like Geerling, feel equally sad / angry / frustrated, but merely say "Please for the love of all that is good, be nice and follow an honour code.".
They dragged the term through different phases, but that’s just projection of will. Theft is undefined for objects with .copy() interface. It’s still there when you look at it.
People have to adjust expectations, not laws. Computers replaced computers, now voice acting replaces voice actors. Your popularity doesn’t mean anything really and wouldn’t it be unfair if only popular could spare their jobs.
> Computers replaced computers, now voice acting replaces voice actors.
It's incredible what web development does to someone's ability to communicate ideas.
Looks perfectly clear to me.
Was that ever so?
According to Google Ngrams, "computer" was not really a term until, predictably, around 1950, even though the verb "compute" has been in use throughout history (it comes from the Latin "computare", apparently).
That tells me "computer" was never a person, always a machine.
https://books.google.com/ngrams/graph?content=compute%2Ccomp...
https://en.wikipedia.org/wiki/Computer_(occupation)
The first reference in the Wikipedia article is to the original definition in the Oxford English Dictionary. Dating from 1613, it refers to a person.
> From 1953 to 1958, Johnson worked as a computer,[23] analyzing topics such as gust alleviation for aircraft.
On a serious note, please refrain from using “webdev makes dumb” theme here. It’s a beaten cliche that does nothing good to any forum. Have a nice day!
In other words, that's just the normal lifecycle of words in a language with an active speaker community. In any stage of history, the meaning of words is just the speaker community's projection of will.
Best I can do now is acknowledge that what counts as "theft" is a complicated topic and can't be decided by a binary "is said object still there after alleged theft has occurred?". I've benefited from some digital theft, naturally, so I might be biased to uphold my own morality but the kind of theft contemporary AI tech has enabled is something else entirely. Somewhere there is where I draw the line.
Recently, I introduced a few friends to the works of digital artist wlop. The immediate reaction was "Is that AI?". I can't help but feel offended in behalf of wlop. It doesn't help that they have made LoRAs out of his work. It's not so much the "theft" of techniques/concepts/etc. that enrages me but rather, the theft of credibility that a human is capable of this output. I imagine Jeff Geerling (and, to a lesser extent, maybe ScarJo) is enraged along similar lines. In this AI summer, other people are fighting for their livelihoods, other people are fighting for their credibility. And, of course, there's an intersection of people whose credibility is their livelihood.
Note that in reframing it as theft of credibility, the owning party has been definitely injured to an extent. As in, said object (credibility) is no longer what it once was after alleged theft has occurred.
And I'm not trying to state some Universal Truths that I will debate to death. Again the whole point is that what counts as "theft" is a complicated topic. I'm sure if you spend a bit more brainpower, you can find analogies that will make me look like a hypocrite. I'm just seeing this community lately strongly signal towards preserving some "original" meaning of words in the belief that it will solve some problem or another and I'm tired of it; I have similar linguistic thoughts about the whole uproar on the term "hallucination" but that's for another comment thread essay.
> People have to adjust expectations, not laws.
I know this thread is about theft but this attitude is downright dangerous in general. People should expect laws to adjust, lest they become irrelevant. Quick example: it's not fair to tell workers to adjust their expectation in light of the emergence of the gig economy. Should they just expect their labor to be exploited then, moving forward? I say, absolutely not. Legislation should catch-up to uphold/strengthen labor laws. Replace "gig economy" with "AI" and we are sort-of back on topic.
Laws are codified expectations.
I understand most of your points, but my main question still stands: how do we choose who’s worth sparing and why it should be Ms. Johanson or wlop or my grandma whose ability basically became stolen(“”?) in the 80s.
I believe this has to do with emotion more than anything else. AI theft is closer to the skin than any worker/engineer who was replaced long ago. But at the same time it is another step in human development. We can decide now if having a nice face and/or talking in a specific way is worth being a job or is a worthless skill. Still quite a skill! Just worth not much, akin to adding lots of numbers in your head now.
Whatever we choose, the non-forward looking choice will be crushed by reality, as usual.
Copyright seems to always have one or another wild wild west going on. Maybe you are in the wrong place if the world constantly jumps and kicks from under you trying to throw you off?
> One can't help but wonder what theft even means any more, when it comes to digital information.
[0] https://x.com/alexeheath/status/1823873344133062680
[1] I mean he said you should legally steal things... whatever that means...
For this kind of misuse, the person needs to have some fame, or it's not interesting to steal their voice. In such cases, their fame can be used for retribution. E.g. I can't imagine that this will be good for the reputation of Elecrow in the end. Next time I read the name of this company, I'll think oh it's that company that is scamming people, not good for them.
I am more worried about the cases where someone uses this to e.g. get rid of a they don't like. E.g. imagine some university lecturer that has done nothing wrong, a student is not happy with their grade, use voice cloning to imply that the lecturer said something that could get them fired. With voice cloning getting really good, how can someone like that defend themselves? (Until this becomes so commonplace recordings are not trusted anymore.)
This can still be very useful when used against non-famous people e.g. in a bitter custody dispute by one party to besmirch the other.
Theft requires the loss of benefit of the stolen object to the victim. Copy & paste just blows over the house of cards that is the system which threatens people with cages and poverty if they use the claimed meme and not pay. I will jury nullify all copyright infringement cases I end up on, where the defendant is human and not a corporation.
We’ve had people who are skilled voice mimics for ever, and they mostly exercise their skills for comedy/satire, and not for misrepresenting people’s opinions. IANAL either but I guess this is based on solid legal grounds, and misrepresenting people would be relatively easy to deal with legally.
I guess the difference is democratisation - we’ve moved from very few people having this skill, to virtually anyone with a computer being able to do something similar. And so policing it will be much tougher, and likely beyond the means of someone like Jeff Geerling if it would require legal action to remedy.
Computers made graphic design approachable, but early adopters oversaturated the market before it stabilized. We’ll eventually figure out social norms and regulations for AI voice mimicry too, but there will be chaos first. Also, tech always moves faster than law. By the time courts catch up, this will be old news.
Question is: who is to say how much is needed before it escapes likeness theft? The king of generic nerd voices is going to claim excessive likeness and the accused lifter isn't going to reveal his whole process. Also tuning AI voices by ear is surely possible soon so category kings are not saved by demanding to be left out of training. A ministry of voice authority sounds bleak.
It's really shitty that they ripped off his voice using AI. In addition, I dislike the trend of using text-to-speech for YouTube videos in general.
You are crazy.
In about 5 years AI voices will be bespoke and more pleasant to listen to then any real human: they're not limited by vocal cord stress, can be altered at will, and can easily be calibrated by surveying user engagement.
Subtly tweaking voice output and monitoring engagement is going to be the way forward.
While AI voices will aesthetically be indistinguishable or even preferable they aren't going to carry any reputation or authenticity, which by definition is scarce and therefore valuable. In fact they're likely going to matter more because in a sea of generic commodified slop demand for people who command unique brand value goes up, not down. That's why influencers make the big bucks in advertising these days.
The fact is, Elecrow's a company I've worked with in the past (never signed any contracts, but reviewed a product of theirs 4 years ago that they provided). They're active in the exact same space my YouTube audience is (Pi, microcontrollers, hobby electronics, homelab).
There are a number of potential Elecrow customers who also subscribe to my YouTube channel (one of them alerted me to the tutorial series, in fact), and I would rather not have people be confused thinking I've sold my likeness or voice to be used for corporate product tutorials.
Especially any competitors to Elecrow, who I may have a relationship with, that could be soured if they think I'm suddenly selling my voice/online persona for Elecrow's use.
There's already VTubers who's whole visual identity is synthetic. Why wouldn't the same happen in any other space where performance can affect the perception of content, but you can now simply engineer the performance?
Like I said: give it 5 years and you'll have influencers who no one has ever heard the voice of, because they don't make content with their own.
No. Their assets are synthetic. Their identity is not. Their value and identity is in the real relationship and trust the creator builds with the audience. And that is what the company in question here tried to imitate. They didn't copy Jeff for his angelic voice, they copy him for the trust implicit in his reputation, name and 750k followers. It's the same reason Chris Pratt still gets hired to do commercials for millions despite the fact that any acting student would do it for 500 bucks and why openAI tried to copy Scar Jo in particular.
All these performances can be done by anyone for nothing, but they have commercial value because of the reputation of the person that's lending their likeness to them. And that's of course also why this is blatantly illegal by the way, we have personality rights and you can't shill your product with someone's likeness without consent, whether that likeness is AI generated or reproduced in some other way.
There is not enough voice space to accommodate everyone. Authors would like to fence off and own their little voice island. For every voice there are thousands of similar ones.
Now I can sit at a computer and spend a few hours doing it.
You are completely blowing past the facts of what a person's voice is, and the value it has if that person is someone special to a lot of people. Are there people out there who can do spot on Obama impersonations? Of course. Is the former President not wanting someone who sounds like him to pretend to be him to endorse a local candidate trying to unfairly "fence off" too much of the "voice space"? Give me a break.
It's 100% about impersonation. We've already seen AI voices get used for impersonation countless times.
Because, in a typical fashion of what AI proponents miss, the voice itself doesn't matter. What matters is the human quality the voice represents! If a voice sounds like person X and person X is trustworthy, then I (either implicitly or consciously) will believe the product they're endorsing is probably good.
This is why we get famous people and experts in commercials. This isn't new, we can just do it without any honesty now.
But aside this nostalgic-ish specific context, I don't see why wouldn't they just create a synthetic voice to begin with it.
They offer different voice cloning techniques today, starting from 30 seconds of audio input (sounds somewhat like the cloned voice but definitely not exactly the same) to multiple hours of voice input (sounds like the actual person). In addition, you can adjust the voices with a few parameters or simply create one by defining parameters.
The voice from the video could be an 'instantly cloned' voice based on a few seconds of voice input (judging from the quality). If you want to do y more advanced clone, you have to proof that it is your own voice.
But we know it does matter - i.e. there's research which shows a good sound quality on a voice call improves whether people believe what you say[1].
Now in any individual session, you probably can't make particularly big alterations, but imagine say, Google or Amazon shipping a modified voice assistant voice as "the default" with every new speaker box? Whether people ask for the default voice, or change it, would all become data which tells you what people are responding to. And so right there, your new "voice of Google" or "voice of Amazon" you use in other places now becomes informed by wide-scale testing of whether people listen to it.
And that's presuming no one simply runs studies where they stick people in fMRI machines and play them an AI voice recording which they module according to neural feedback till it's "optimal".
[1] https://today.usc.edu/why-we-believe-something-audio-sound-q...
> training <
My prediction is that to get to that point you need to hire so many editors that you may as well have just hired the VA outright. We'll get more fronts in the meantime not too dissimilar from the Amazon Checkout debacle.
I believe the point here is to litigate it before it can just freely synthesize 100 voices it stole without compensation.
We've been able to product "voices" for decades. The issue isn't the tech so much as its training set.
I think this matters more in the court of public opinion than in real court in both cases though.
Same as it happens with unauthorized use of someone’s images. And platforms and their moderation teams have processes in place to report and remove that. Looks like we need something similar for voice.
You can absolutely positively find a free lawyer if your issue is interesting enough.
This is the most interesting issue of our day.
https://techcrunch.com/2024/09/19/here-is-whats-illegal-unde...
Not sure if those laws apply to Jeff tho, as they concern porn, politics and employer contracts.
Make a video, say what you think, get views, and probably put more pressure on Elecrow to respond.
It was linked in the article.
Does this controversy all become free publicity for elecrow?
Since that guy was CEO of Google it’s all good right???
https://www.theverge.com/2024/8/14/24220658/google-eric-schm...
We definitely need to overhaul a lot of these white collar fines. I just watched a video today and learned the federal maximum fine for being caught using child labor is capped at $15k per worker. No wonder child labor has skyrocketed over the decade.
It looks like we're heading in that direction.
[1] https://old.reddit.com/r/redscarepod/comments/1fmiiwt/which_...
IANAL and not sure about regional precedence on these topics, but there are plenty of ads where lookalikes or voice actors are used to use someone's likeness. they are mostly in satire, but there is yet to be a case where there was a litigation over this or prior approval needed.
we have ai-based voice abuse in the political sphere, and where there was only one legislation for banning the use in voice calls for one country (https://news.ycombinator.com/item?id=39304736), another country actively used the same underlying tech to aid their own rallies (https://news.ycombinator.com/item?id=40532157).
the tools are here to stay, but what is fair use needs to be defined more than ever.
Satire is one of the few use cases of fair use that hasn't been torn down. So that tracks.
> there is yet to be a case where there was a litigation over this or prior approval needed.
There's quite a few over impersonation. Most broadcast media knows how to skirt the rules though.
Although it was not too hard to create I believe making it easier is something i don't like to achieve...
I hate to say this but ruining a narrators existence with AI seems to get easier every day.
Regarding how easy it was to clone my favorite narrators voice with open source tools I'm a bit afraid of what amazon could do with a whole cloud and massive man power
She has/had two numbers; magic jack and google. When I tried to call her, the magic jack was no longer in service and google said something about "unavailable".
I reached out to my cousin (my aunt's daughter) to inquire. I was told her number (and perhaps other things) had been "hacked", whatever that means. She had recently broken her hip and was in a hospital recovering.
With this on my mind, I received a call (from the google number), strangely, while processing files with GPT. My skepticism was primed and ready, possibly making me paranoid. However, I did my due diligence and asked dozens of questions, mostly boring things that she typically wouldn't have patience for. Sometimes she'd reply with a reasonable answer and sometimes not, which made it difficult to evaluate. Toward the end, I asked where she was. She said, with an awkward tempo "I'm at home, in Cuenca", which I found odd because she'd normally just say she was at home, period. I then pressed her to tell me where she was before she returned home. She said she didn't understand. I rephrased the question, stating that it was a simple inquiry, eg "where were you before going home?" She said "this is getting too strange and confusing " and killed the call.
I notified my cousin, telling her I thought something was suspicious, still cognizant of all the characteristics one would expect from a 90 year old recovering from a serious injury. My cousin might, technology wise, be in AOL territory.
About 5 days later, I received a call from my aunt, on the google line. This time,I was more passive and cautious, but again, asked dozens of boring questions to probe the situation. I was surprised by both her ability to answer certain questions and also her inability to answer some questions. I tried to ask questions on topics we'd never discussed, in case the line had been tapped for a long time and referencing was established by an imposter. I had begun to suspect I had been paranoid. But several aspects were burning me: 1) typing noises in the background 2) Shatneresque pauses for nearly every reply 3) refusal to answer some specific questions.
At the end of our apparent conversation, I asked her to do a very serious favor for me: send me a selfie, with one hand making the thumbs up gesture. She replied "I'll send you a photo of my passport ". I replied "that's stupid, ridiculous and serves no purpose. Don't do that. Understand? Do NOT send me a passport photo. I'm asking you something very important. Do exactly what I asked. Will you do this?" Her reply: "yes. What is your email address?" This was odd. I told her she already knew and it's the same one she'd had for years. She asked that I tell her anyway. Ok, 90 years old, traumatic injury, possible prescription drugs... "It's my full name @ xyzmail com". We killed the call.
I immediately called my cousin and told her of my suspicions, including some my aunt's babbling about all her finances and accounts being inaccessible. She said that was strange because she just deposited 8k into her account. Meanwhile, a notification appears in the phone, an email from my aunt. It's a photo of her passport.
Having no authority in this situation, but plenty well annoyed, I immediately jumped on a real computer and ran the photo through exiftool. The photograph was taken in 2023 and it was August of 2024. I then grabbed the geo coordinates (cryptically presented in exiftool) and with some effort, geolocated the image to right on top of her former residence, in Cuenca.
I still don't know WTF is going on and my cousin thinks I'm a dingbat. But what I know for sure, is this is an age where such things are plausible enough and will soon be inevitable. The way I think may be deranged, but I truly don't even know if my aunt still exists. But I can have a pretty compelling conversation, either with her, or something strongly resembling her, minus the Shatneresque pauses, typing noises and selective amnesia.
Or go there for a weekend and check?
For example, if you asked for a selfie, she might just think you want a picture of her, and she remembers that she has a picture of her that she took last year where she looked good (passport phot that people dress up for), and wants you to have a good photo rather than one where she looks miserable in a hospital.
The way you tell the story makes it sound suspicious, but next time I would just be direct and tell her something seems suspicious to you, that someone could impersonating her, so that is why you are asking.
If someone is targeting you, chances are they already saw your comment here so that hypothetical person already know you're on to them, so saying that on the phone won't give any new information away.
I couldn't include our entire dialogue into an HN comment, but yes, upon prodding as deeply as I could and running out of ideas, I explained my suspicions. The response wasn't what I expected, but not direct evidence supporting my concers quite either.
If it was my aunt, she understands well. If not, the perpetrator does too.
One of a few other instances which got my attention was a voicemail she left, which I retain a recording of. It starts by saying her name, awkwardly, followed by a 5-8 second pause, then saying "Hi. This is <her name>. I always refer to her by her abbreviated single syllable name, while the voicemail used her formal, full name.
I haven't heard from her since saying that if anything went wrong, I'd be looking for fingerprints on the passport.
For example, if you asked for a selfie, she might just think you want a picture of her, and she remembers that she has a picture of her that she took last year where she looked good (passport phot that people dress up for), and wants you to have a good photo rather than one where she looks miserable in a hospital.
The way you tell the story makes it sound suspicious, but next time I would just be direct and tell her something seems suspicious to you, that someone could impersonating her, so that is why you are asking.
If someone is targeting you, perhaps are they already saw your comment here so that hypothetical person already know you're on to them, in which case saying that on the phone won't give any new information away.
Maybe just ask the cousin not to send any more money?
Regulating prolong adoption and take resources.
I don't have a dog in this fight but just to be clear, OpenAI has stated that they paid a voice actor to create the voice ("Sky") that sounds like Scarlett Johanssen. There was no "cloning" or "stealing" (that they say).
https://openai.com/index/how-the-voices-for-chatgpt-were-cho...
They may have been absolutely fine if Altman never approached Scarlet. But context matters.
Is that some sort of a coat of arms?
I think we even did a thing in one of my art classes on heraldry once, and the idea was to try to figure out what symbols you would choose for your own personal crest, based on what kind of things we'd see in historical ones.
1. Why clone Jeff's voice?
When I was messing with stable diffusion using Automatic1111's interface, I noticed it came with a big list of artists to add to the prompt to stylize the image in some way. There was a big row in the media about ai art reproducing artists work and many artists came forward feeling it was a personal attack. But... I mean the truth is more general than that. When I pressed a button to insert a random name into a prompt, my goal was not "yes give me this person's art for free", it was "style this somehow".
I wasn't personally interested in any particular artist, I honestly would have preferred a bunch of sliders.
Jeff here is clearly a good speaker. That's a practiced talent and voice actors exist because it's hard. Elecrow wanted a voice over and they wanted it to be as good as they could make it. Jeff is very good. So did they want Jeff?
I think what they really wanted was a good and cogent narration with the tenor of a person. Not a machine making noises that sound like english. If they had an easy way to get that, we wouldn't be talking about it here.
2. What function does copyright serve?
Well. I think a reasonable argument would be that if people were able to reproduce your work for free, you would quickly find yourself without a monetary incentive to make more of it.
So. What happens if you combine answer 1 with answer 2?
I think it leads to: "We should consider making it illegal to automatically reproduce the work of an artisan.", you know, the luddic argument. An argument that has been perceived to be, more or less, settled.
So it seems to me: That for individuals, harms matter, and for society, it doesn't.
I think the general assumption is that they wanted to, at the very least, strongly imply his endorsement of the product or video.
Which I would say they did effectively. If I had happened on a clip of one of these videos outside the context of this controversy, I could have easily gotten the impression he was working with the vendor.
yeah and that's the problem. The style of an artist is a developed thing. To think that one could borrow your style not through learning and caring, but through mathematically analyzing the width, and colors, and patterns and applying it to a random noise — that's kinda insulting. If nobody cares about my real work, why do they care about using my style, then? Develop your own an teach your AI on that, if there really isn't any difference.
People say that AI learns how a human would. But a human wouldn't (couldn't!) learn like an AI can. He can't look at the pixels, can't mechanically churn through patterns. If someone can learn from art like AI learns art, I would also be opposed to them learning anything from me :D
Jeff has worked with and endorsed some of their products before, so that puts a wrench in that theory of "well they just picked a clean voice" and makes this almost litgable.
>I think it leads to: "We should consider making it illegal to automatically reproduce the work of an artisan.", you know, the luddic argument. An argument that has been perceived to be, more or less, settled.
There's the labor argument: People who's voices are samples should get a residual on the product they are being used for. Combine that with some sort of lack of liability on the subject when AI is used and we'd have a win-win.
But that requires money and companies don't want to pay other people. So we come at an impasse that leads to the luddite argument. Take the ball and go home if you don't want to pay. The fact that this comes into so few people's minds shows how successful companies are at casting off the idea of residuals.
If someone cloned both Shaq's voice and Jeff's, and used them to endorse sneakers - I think it's a fair assumption that Shaq would see this as a business risk, and Jeff .. I'm going to go out on a limb, and assume he'd probably find it hilarious. Using Jeff's voice for sneakers would be more akin to your example of finding a midwestern voice with a useful corpus. Using Shaq's would be a much more obviously targeted appropriation.
What we're looking at here appears to be exactly this scenario, except this is Jeff's niche, not Shaq's. Using Shaq's voice for SBCs and related products would feel quite absurd - using Jeff's feels like a much more obviously targeted appropriation.
Most likely all existing youtubers will have complete voice and video digital clones made out of them. Then you can also tune an LLM on their scripts and it'll respond in the same character as well.
In theory you could also bring back ones who are dead, which would be very interesting in a historical sense. Like if we had hundreds of hours of Napoleon talking in front of a camera, it would be trivial to recreate a digital version of him for anthropologic study, maybe even having various figures debate things with each other. That's what historians a century later after we all die will be able to do with impunity.
It's kind of like saying "how do we know that face is similar/identical"? Humans are surprisingly good at knowing when something feels off in other humans. Even if we lack the vocabulary to fully explain the difference.
Luckily, national ID can easily solve that problem offline. It'd be a more interesting problem if some company searched far and wide to identify someone that has a similar face to [famous person]. And that's part of the issue in that there's no way to verify "is my voiced sampled here?". Opening the black box can help with that, but that's something companies are fighting to the death to not allow.
> What is the likelihood of a random AI voice sounding similar to some real online creator?
Given how much data was scraped, less likely than the fact that their data is in fact in the training set.
We already had fake news and organizations willingly spread fake news.
We had clearly fake pictures and people believing that.
Flat-earthers, no-vax and whatever.
This is just another brick in the wall.
There is absolutely zero evidence for this. I find it infuriating that this keeps being stated as a fact. So they go and hire a voice actor and clearly use her voice to train, but then they also scrape Scarlett Johansson from youtube and splice it into the training data to make the voice a bit more like hers? Really does that sound realistic?
Motive: Altman had some weird boyish thing for her and they asked her first, she said no.
Means: Lots of available data to use from her movies. They probably trained a model first without releasing it just because it’s ridiculously easy. Especially for OpenAI.
Opportunity: AI is astonishingly good at laundering and remixing without exposing the training set, for previously-unseen levels of plausible deniability.
They just about manage to make a good multimodal transformer that can generate audio and you expect that right away they can also interpolate in latent space? How does that actually work? It's not so simple. What benefit do they have from training on Scarlett Johansson's data, because they sure as hell have a big risk. They clearly hired a voice actress and they clearly told her to sound like Scarlett Johansson in "Her", and the end result perfectly fits with that. The voice doesn't "uncannily sound like SJ", no it just vaguely resembles her voice and mostly just mimicks the mannerisms from the movie. For me this is a perfect example of Occam's razor. One explanation is simple and realistic. The other explanation requires significantly more advanced AI control than OpenAI has claim/demonstrated, and it requires Altman to be so obsessed with the SJ idea that he goes out of his way to secretly train on her voice, risking legal exposure, while still hiring a voice actress.
> they can also interpolate in latent space? How does that actually work?
I don’t know. I assumed it is the case because of the vast and excellent interpolation AI appears to do in other domains.
Sure it's plausible just like many other forms of baseless speculation are, I still don't think to state it as a matter of fact like it is happening in this and other articles and in comments.
> I don’t know. I assumed it is the case because of the vast and excellent interpolation AI appears to do in other domains.
Yes but do you know of any examples with sequential data, and in particular transformers? I don't and I think it is a somewhat different thing. With pictures you just train an encoder into some low dimensional latent space and there you can interpolate, but working with sequences you don't want to give the whole sequence as an input to the model and generate the whole sequence as an output.
The natural way to do interpolation with an audio transformer model, which it can in principle indeed do, would be to somehow audio-prompt engineer it to speak in a voice that interpolates SJ and your voice actor. But that would be a pretty fragile method.
Except that never happened and the voice belonged to a completely different voice actress and Scarlett Johanssen had exactly zero right to prevent this person from making money as a voice actress lending it to AI.
These complaints remind me a little bit of the story that a man complained that his photo was used to illustrate the article about how all hipsters look the same and it eventually turned out it wasn't his photo.
She said no.
So they found a soundalike and Tweeted out references to the movie Her (starring Scarlett Johanssen as an AI chat bot) in the days leading up to launch.
Scummy as fuck from OpenAI regardless of the technical legal rights and issues involved.
If there were just two blonds in the world, one famous and the other not and you wanted a blonde actress for the role and she said no is it scummy to hire the other one?
Is it scummy to hire "discount" Matt Damon instead of Matt Damon?
It's illegal, you can't hire an impersonator and allude to it being the real deal. That's the issue.
> Is it scummy to hire "discount" Matt Damon instead of Matt Damon?
If you imply you've hired Matt Damon by tweeting something about Matt Damon and your "discount" had a profession as a Matt Damon impersonator, yes. Yes 100%.
This is illegal with or without AI. We've literally had these laws longer than the tech.
If Matt Damon said no (e.g. leaving a paper trail) and you want people to pretend it's Matt Damon anyway, yes. Look up Ford v. Milton. You can't just go behind an artist's back like that.
If you want to argue about soundalikes in spinoffs, those are discussed in contracts (and those kinds of figures are making 7+ figures as is, so it's fair on them to choose to let them do that).
/s
How?
However, in this situation, the right of publicity is probably more applicable.
In order for this to be a FALLACY, you have to explain how an appeal to law doesn't work here. Simply appealing to law itself isn't a fallacy.
Describing a reasonable legal principle in terms of physics phenomena does not make it unreasonable.
This is a good thing, even though your comic is nothing but a bunch of JPGs--digital patterns--that can be copied infinitely without depriving anyone.
Voice-reproduction rights can get complicated in the AI age, and I don't claim to have the objective moral truth. But there are very good reasons to give legal protection to the creators of digital patterns, as a general principle.
"If a product is being sold that predominantly exploits the commercial value of an individual's identity, that product should be held to violate the right of publicity and not be protected by the First Amendment, even if there is some "expressive" content in it that might qualify as "speech" in other circumstances."
Whether or not the voice is determined to be predominant would be for courts to decide, of course, but there's clearly an argument.
1: https://law.justia.com/cases/missouri/court-of-appeals/2006/...
If you argue similarly, then the whole juridical system is nonsensical, because everything is just particles and waves, and different configurations thereof - not to mention the many protected things that are acts, which are neither particles nor waves, and are completely made up.
I'd say it's desirable to regulate something like this, however nonsensical-seeming, so that we can at least somewhat protect the individuals, and the general well-being of society.
I do understand rivalry in economics, but that's not the issue here. The issue is the need to protect individuals from abuse, part of which is the one's right to their own likeness. This likeness is not free for the taking. Now, we can call this owning the likeness, or we can call it something else, but having something, that is not free for taking, sounds an awful lot like owning something.
Social/legal constructs like owning ideas or the sound of a voice exist not because they make any sort of logical sense, they exist to give privilege and power to certain groups at the expense of others.
Social constructs don't not exist just because they are social constructs. The physical world not the extent of reality. Social constructs are also real because we make them real as a part of our own human experience.
It implies identity, and control. The former being most relevant here. And yes, if you want to use someone's identity you need to at least get consent.
>Exclusive control over a scarce resource.
"Scarce" is doing a lot of heavy lifting. And in the grand scheme, everything can be "scarce". The internet isn't infinite either, only in a virtual sense (no pun intended).
Meanwhile, thoughts and data are resources, and you can own the idea behind it. Hence, "intellectual property".
> there is no feasible model that explains why one person has control or ownership of it over another.
That's where identity came in. If an inventor got run over tomorrow, there may still be iterations on the invention, but it would completely alter the course of iteration. That's ownership in a sense. Ownership that your ideas can't just be understood or replicated unless you transfer your thoughts to another medium. And like any conversion, data is lost in the exchange.
>they exist to give privilege and power to certain groups at the expense of others.
Yeah, the less off inventors. Do you know why IP is automatically granted? Because someone unaware of the legal channels would just have their own IP signed off once anyone in business gets a whiff of it.
of course, the priveledge still abuse it. But when everything becomes a free for all, the rich get richer.
It implies whatever society decides that it implies, no? For example, many societies had or have conceptions of land ownership that are dramatically different from the western model, oftentimes to the extent of not having individual land ownership at all.
> Social/legal constructs like owning ideas or the sound of a voice exist not because they make any sort of logical sense
They don't make logical sense to you. Consider that for a very long time in the US the idea of owning people made logical sense to the majority.
> they exist to give privilege and power to certain groups at the expense of others.
Do not almost all forms of ownership do this? Some people own hundreds of thousands to millions of acres of land that I have no right to access. Massive swathes of the surface of the planet that we all share. Is the ability to own land not a far greater privilege than the ability to protect your voice from being cloned without your consent?
Are you trying not to be taken seriously?
What happens when this "AI" is used to sway an election?
Last month a family got hospitalized after eating mushrooms they found and identified from a AI generated book. They didn't know it was AI generated. What happens when and this is not if, alas but when people die from this?
This shit is a danger to democracy and human lives. Napster was not.
Yeah, mushrooms are very dangerous, but the argument here is that the book could be written by a human. So what's the best way forward? Ban A.I. books, and human books?
A.I. also, is the best way IMHO to identify mushrooms. Mushrooms are drastically different from one another by magnifying their spores using a microscope. That's not the case when looking at the fruit. Mushrooms totally unrelated to one another, may turn out to be very similar, depending on season, humidity, rain, elevation, tree hosts, temperature and soil.
However when a human tries to examine the spores, he has to compare the spores to thousands of mushrooms to be sure. That's an amount of information, that only mechanically could be tackled effectively. A.I. may have a good chance to solve that problem.
Also, I'd want some kind of verification before using the mushroom app. Who made the app? Do they have people on their team familiar with mushrooms? Botanists or whatever you call them. Some random dude with free time and access to maybe 10 species of mushrooms in his backyard.. even if his intent is good, is still dangerous.
https://hachyderm.io/@inthehands/112006855076082650
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
They're already trying to use AI for breast cancer scans and what not, I don't see why we can't do it for mushrooms.
I don't understand why so many people in these discussions are so keen to ignore the impact of scale and accessibility when it comes to new technology, and specifically this new technology. Yes, most dangerous things that can be done with AI can also be done by humans.
Is it not meaningful that these dangerous things can now be done far more cost effectively? It would've made no financial sense in the past to spend hours/days creating a fake mushroom identification book. You'd almost certainly never get enough sales to make it worth it, especially once people realized your book was nonsense and potentially dangerous (getting you delisted, as it seems like the seller did with this book). Now you can just ask an AI to "generate 100 book ideas, scripts, and images." Who cares if almost all of them make very little money when the time and $ cost to creating them is near zero (it looks like this book was physical but especially for digital books, videos, etc.).
Is it not meaningful that dangerous things can now be done (or may soon be able to be done) by more people with less skill? The time/money investment required to learn skills with the potential for destructive use is IMO a strong filter against the people who would do those destructive things in the first place.
OpenAI and I think Google have talked about making intelligence "too cheap to meter." I'm not sure that is a good thing. I could be convinced, but every poorly thought out dismissal of AI dangers only makes me increasingly sure that we aren't ready for it.
For those who are less concerned about AI dangers, consider maybe that premature reckless deployment of AI also has serious potential for generating social backlash that might end up slowing or halting the adoption of AI for positive purposes. The average non-tech person I know thinks of AI as a useful improvement over Google, and maybe thinks AI generated images are cool, but they aren't in love with it. It could disappear tomorrow and they'd shrug. In fact, public sentiment against AI (in the US) seems to be rapidly increasing, and is bipartisan [1]. If the promise of AGI is achieved this will go into overdrive should (when?) mass-unemployment happens. Look how far right-wing politicians are able to go by drumming up fears about immigrants stealing jobs.
https://www.pewresearch.org/short-reads/2023/11/21/what-the-...
It is meaningful. Humans can publish 10 books with 10 mistakes inside, A.I. can publish a trillion books with countless mistakes. Radically different scale.
Imagine this: A writer after he finishes writing a book, he goes to a government office, and stamps electronically the hash signature of the book as "Human Approved". Then anyone can search the Book stamp in an internet database. Everything else is labeled as A.I.
Is it so difficult for a government to use a software like that? How about an open source software? (It doesn't exist, but it could be implemented by some people.)
> In fact, public sentiment against AI (in the US) seems to be rapidly increasing, and is bipartisan
That cannot be true. A massive poll of at least 10.000 people could demolish that myth. The poll should be organized by the age of participants, rather than political parties. Candidate ages are, 18-30 years old, 30-60, and 60-Rest.
Ages 18-30 will be 80% in favor of A.I.
Not sure what the point of this would be, the government worker will almost certainly have no ability to verify that the books people bring in weren't written with AI.
> That cannot be true. A massive poll of at least 10.000 people could demolish that myth.
The third link in the pew article I provided has some methodological details:
"Pew Research Center conducted this study to understand attitudes about artificial intelligence and its uses. For this analysis, we surveyed 11,201 U.S. adults from July 31 to Aug. 6, 2023."
> organized by the age of participants
I agree this would interesting. I found a smaller survey (n=1500) that suggests while young people are less anti-AI than older people, they are still heavily anti-AI [1]. Only 20% of Gen-Z states they're "excited about AI," while 29% say they're skeptical and 18% say they don't trust it. This meshes with my experience; even among colleagues that use and/or research AI they're nervous about the impact it'll have on society.
> Ages 18-30 will be 80% in favor of A.I.
I'm curious why you would assume this. In scenarios where AI causes major job loss, young people have by far the most to lose, as they haven't had nearly as much time to build up resources and acquire property.
What do you mean "when"? It's already happening.
https://apnews.com/article/trump-taylor-swift-fake-endorseme...
It already happened and the world went merrily on its way towards wherever the hell it goes.
I don't necessarily agree because money has a lot to do with it. In theory, anyone can use AI for evil. In practice, only the richest can inflict damage. It's a matter of scale.
That's why me killing someone is pretty bad, but me building a Nuclear Warhead is much worse. Luckily I don't have the power to build a Nuclear Warhead. Now anybody with enough money has the tech to just make one.
That's always been true for every technology ever. Even nuclear warheads you mentioned. They are not controlled by poor people. Even hacking that was seen as sort of equalizer is way more dangerous in hands of states that can hire whole armies of hackers.
This guy doesn't have 1/10000 of the relevance Metallica had at the time
It's weird that you seem to think a person's level of fame is relevant to a discussion of their legal rights.
First off, this reddit speak needs to go in a fire and die. Seriously, stop typing like this, nobody likes it.
Secondly, yes voice imitators CAN be criminals. If they attempt to or allude to being the real deal. You can't get on a phone interview with the news and say you're Barack Obama, even if your impression is really good. Yes, that's illegal and has been since before AI of this form existed.
Which is not the case in discussion, there is no impersonation.
> First off, this reddit speak needs to go in a fire and die.
Are you ok? I think you should calm down before posting.
The trend has been going against copyright ever since internet was invented. We used to go for passive consumption - radio, TV, books, print magazines. But now that age has passed. We have changed. We prefer interactivity - games, social networks, web searching the billions of contents online, youtube clips commented and shared around. In this age copying is like speaking.
Now comes AI and pushes this trend even deeper - more interactive, more remixing and adapted to the user. We should take a hint - the idea of owning content doesn't make sense anymore. Any content can be recreated in seconds now, or similar content was already posted online years ago. Protecting expression is useless and protecting styles would destroy creativity. Quite a catch-22.
We should take a look instead at new models that popped up in the last decades - open source, creative commons, Wikipedia, open scientific publication. Maybe we need to decouple attribution from payment, like scientific papers, they cite each other without monetary obligations. In social networks comments respond to each other, it would not work if we had to pay to read. Even in real life, people don't pay to hear others speak, and are reusing ideas without issue.
I am aware this sounds bad for copyright, but I am trying to state the reality, not propose my own views. There are deep reasons we prefer interactivity to passive consumption. Copyright was fit for a different age.