DoNotPay has to pay $193K for falsely touting untested AI lawyer, FTC says
arstechnica.com
arstechnica.com
Wow, that's brave. Create a wrapper around ChatGPT, call it a lawyer, and never check the output. $193k fine seems like peanuts.
Sometimes I think about where I would be in life if I had no moral or ethical qualms. I'd probably be running a company like this.
>[...] DoNotPay's legal service [...] relying on an API with OpenAI's ChatGPT.
Perhaps they rolled their own chatbot then later switched to ChatGPT? Either way, they probably should have a lawyer involved at some point in the process.
Idk why they needed to have a lawyer involved though. Many processes in life just need an "official" sounding response: to get to the next phase, or open the gate to talk to a real human, or even to close the issue with a positive result.
Many people are not able to conjure up an "official" sounding response from nothing, so these chatbot/ChatGPTs are great ways for them to generate a response to use for their IRL need (parking ticket, letter to landlord, etc).
Machines are also not allowed to be the "author" of court documents if they actually get to court (so far as I'm aware). A lawyer has to sign off and claim it as their own work, and doing so without the lawyer reading it is pretty taboo (I think maybe sanctionable by the BAR but I could be wrong).
Do you think WebMD should be renamed? Lawyers seem to be better at defending their turf than doctors or cops.
To your second point, it's very funny how OpenAI seems to have soured the tech crowd on tech.
In this particular case, I'm not sour because of OpenAI. I am sour because of deceptive and gross business practices highlighted in the article.
It was actually free back then I used it the one time and felt grateful enough for the help that I signed up for one cycle and then cancelled (since I didn't have a continued need for it).
Nothing gross or deceptive.
They just had to pay a fine for deceptive advertising. The article lists a number of other deceptive and immoral business practices.
You read an article about it and 7 years later and are convinced they are crooks.
Actually, you read an article and assumed that your anecdotal experience from 7 years ago is more reflective of how a business operates than a current year investigation into that company by a federal agency.
Nobody is arguing that they deceived you personally 7 years ago.
It wasn't an eighteen dollar a month lawyer, so your description of it as "any other" is wrong.
Using OpenAI as an example. ChatGPT is wonderful for the things it's made for. It's a tool, and a great one but that's all it is.
But OpenAI itself is a terrible company and Sam Altman is a power hungry conman that borders on snakeoil salesman.
And I'm soured on people like the CEO of my company who wants to shove a GPT chatbot into our application to do things that it's not at all good at or made for because they see dollar signs.
Welcome to humanity.
You could blame capitalism, but it happened there too. In fact in every system.
They represent an amplifier for the enshittening that was already souring the tech crown on tech.
LLM's used in this sort of way, which is exactly OpenAI's trillion dollar bet, will just make products appear to have larger capabilities while simultaneously making many capabilities far less reliable.
Most of the "win" in cases like this is for the product vendor cutting capital costs of development while inflating their marketability in the short term, at the expense of making everything they let it touch get more unpredictable and inconsistent. Optimistic/naive users jump in for the market promise of new and more dynamic features, but aren't being coached to anticipate the tradeoff.
It's the same thing we've been seeing in digital products for the last 15 years, and manufactured products for the last 40, but cranked up by an order of magnitude.
It's exhausting and disheartening.
All engineers I know in tech are bashing AI left and right and going home to work on AI projects in their free time.
* Is massively overhyped by certain persons and companies, and
* Is both interesting and has loads of promise, so is worth working on in your own time
They aren't contradictory at all.
Meanwhile, I dunno, I have some begrudging admiration for the folks getting rich selling premium GEMMs, but eventually they are going to piss off all their investors and cause a giant mess. Like good luck guys, get that money, but please don’t take us all down with you.
I think people love the idea of DoNotPay: A magical internet machine that saves people money and fights back against evil corporations.
Before ChatGPT they were basically mad libs for finding and filling out the right form. Helpful for people who couldn't figure out how to navigate situations by themselves. There is real value in this.
However, they've also been running the same growth hacking playbooks that people disdain: False advertising, monthly subscriptions for services that most people need in a one-off manner, claiming AI features to be more reliable than they are, releasing untested products on consumers. Once you look past the headline of the company you find they're not entirely altruistic, they're just another startup playing the growth hacking game and all that comes with it.
EDIT: yeah, yeah, I hear you. You can survive on VC money, and maximize share price instead of focusing on profit. You can also be a small business owned by good people just trying to make a living, but then you still have to not get drowned out by more ruthless competition. The purpose of a business is not always to maximize profit.
That's false. The purpose of a business is whatever the owners of that business decide. I've known a large number of business owners that chose less profit in exchange for any number of other attributes they valued more than max profit: more of their own time (working less), better serving a local community by donating a lot of resources / air time (media company), paying employees abnormally higher wages (because said employees had been with them a long time and loyalty matters to some people a lot) - and so on and so forth.
Max profit is one of a zillion possible attributes to optionally optimize for as the owner of a business. The larger the owner the more say they obviously will tend to have in the culture.
Facebook as a prominent public example, hasn't been optimized for max profit at any point in the past decade. They easily could have extracted far more profit than what they did. Zuckerberg, being the voting control shareholder, chose to invest hilariously vast amounts of money into eg the Metaverse / VR. He did that on a personal lark bet, with very little evidence to suggest it would assist in maximizing profit (and at the least he was very wrong in the closer-term 10-15 year span; maybe it'll pay off 20-30 years out, doubtful).
The pursuit of max profit is a cultural attribute, a choice, and that's all. It's generally neither a legal requirement nor a moral requirement of a business.
So, in a pretty real way, there is a legal requirement, though like many such things in the United States today, it is not properly formalized.
This is quite an important distinction because it implies we may want to limit the prevalence of those things and increase the prevalence of small businesses and privately-held medium businesses that can advance other societal goals.
The government has consolidated around the current set of incentives, both directly through its own arms and indirectly through legislation and policy, leading to the result we see today. Breaking these large businesses up may lead to some disruption for awhile, but if the incentives stay the same, the same end result will likely be arrived at again before too long.
So no, profit is not a universal motive. But it's a popular one; if you have a conventional business and you expect to stay solvent year-over-year, then you make profits, you stay in the black, yes? Nobody can prognosticate when the lean years will come, and so you watch that bottom line and keep as much cushion as possible, to ride out a bad year or two.
Furthermore, if a business is competing with other businesses, that's going to moderate the profit motive with market share and other considerations. But I would say that publicly-traded companies have the strongest impetus to profit and satisfy shareholders. The publicly-traded space is far more constrained than other businesses or entities, such as charities, public interest groups, political action, NGOs, etc.
When is humanity going to start seeing some patches against these exploits? Is common sense still in beta?
In America, it is legal to call your uncarbonated soft drink "vitamin water"!
Lawyers have also led significant efforts to simplify the law. For example the American Bar Association has consistently created simple model statute frameworks that eventually are adopted.
Law is complex because society is complex.
"Simple" is not an accurate description of typical model legislation.
> Law is complex because society is complex.
Law is complex because it's an evolved system influenced by politics and corruption. The extent of its complexity is not intrinsic and much of it is specifically a defense mechanism against public understanding, because the public wouldn't support many things in the status quo if they understood the workings of them, and the people who do understand the workings but prefer the status quo use this to their advantage.
Try the example of drafting a standard apartment lease, over millions of transactions between landlords and tenants lots of edge cases emerge. So over time leases get more complicated. And then the law around interpretation and enforcement gets complicated.
Case law is full of politics. How do you think courts resolve the ambiguities? If there was an objective standard for how to do it then judges could be replaced by computer programs. Judges are used instead because rigorous and consistent application of rules would lead to outcomes that are politically inexpedient, so judges only apply the rules as written when politics fails to require something different.
> Try the example of drafting a standard apartment lease, over millions of transactions between landlords and tenants lots of edge cases emerge. So over time leases get more complicated.
This is just a facet of how contracts and lawyers work. The law creates defaults that a contractual agreement can override, so each time the law establishes a default that landlords don't like but are allowed to change, they add a new clause to the lease to turn it back the other way. What they really want is a simple one-liner that says all disputes the law allows to be resolved in favor of the landlord, will be. But politics doesn't allow them to get away with that because what they're doing would be too clear to the public, so politics requires them to achieve the result they want through an opacifying layer of complexity.
> because the public wouldn't support many things in the status quo if they understood the workings of them
Personally, I've observed the opposite more often: somebody feeds the public a clickbait-ey and manipulative "explanation" of how things work, and public becomes enraged without any real understanding of complexities and trade-offs of the system, as well as unintended consequences of proposed "fixes". It is the main reason why socialism is a thing.
The reason we have laws is to facilitate corruption? That seems like something we ought not to want.
> Personally, I've observed the opposite more often: somebody feeds the public a clickbait-ey and manipulative "explanation" of how things work, and public becomes enraged without any real understanding of complexities and trade-offs of the system, as well as unintended consequences of proposed "fixes".
That's the media. The government over-complicates things. The media over-simplifies things.
It has the same cause. People tune out when something becomes so complicated they can't understand it. So if they want people to pay attention to them, they over-simplify things. If they want people to ignore what they're doing, they over-complicate things.
No, the reason we have laws is because politics and corruption and crime is intristic to society. They are the reality of human condition which can't go away and we can't ignore, so we have to deal with it.
> That's the media. The government over-complicates things. The media over-simplifies things.
I had in mind the part of the "media" which "rebels against the media", Noam Chomskies and Michael Moores of the world.
Public corruption is intrinsic to government action, but the way you constrain it isn't by passing laws that limit the public, it's by limiting what laws can be passed by the government.
> I had in mind the part of the "media" which "rebels against the media", Noam Chomskies and Michael Moores of the world.
Chomsky probably isn't a great example of over-simplifying things. Many of his criticisms are legitimate.
But having a legitimate criticism of the status quo is a different thing than having a viable solution.
There are a lot of cases (to the point where I expect your average person sees dozens of them every day) where the media isn't just "over-simplifying"; they're presenting things that are specifically crafted to both
- Be factually correct
- Make the reader leave with an false understanding of the situation
This exact same thing happens with political campaigns.
Over-simplifications are false. They don't even meet the bar of being factually correct, whether because the proponent is willfully leaving something out or because they're ignorant themselves. It's not impossible for it to happen innocently, because people selling simplistic narratives often build a following even when they're true believers.
The thing you're talking about is selection bias. It's the thing assholes do when they want to lie to people but don't want to get sued for defamation. Whenever you discover someone using this modus operandi, delete them from your feed.
Common sense has been deprecated. The system is just waiting for all of the modules with common sense preinstalled to sunset.
Common sense was relegated to a legacy feature status after "clout" and "fuck you got mine" were released.
It will not see further updates and will be sunset shortly. Sorry for the inconvenience.
i think that's called "consequences" and it's the subject of the article you're commenting on
Some people gain enough trust that once they start to exploit it, that there often isn’t a mechanism to apply consequences.
i.e. - politics
If so many people are willing to take the claim at face value, that suggests to me a general naivete and lack of understanding of AI out there that really needs to be fixed.
Aside from AI-related stuff, GGP mentioned "monthly subscriptions for services that most people need in a one-off manner". It's amazing to me that anyone would sign up for a monthly subscription to anything at all, without any consideration for whether they'd likely have a use for it every month.
The term "snake oil salesman" has been around since the 1800s and that's effectively what most of these growth hackers are. I'm sure there are plenty of terms for the same practice of fraudulent marketing that predate that by centuries or even millennia. If you can hype people up enough about what you're selling and get them imagining how much better their live's will be using your product a certain number will buy into anything (in DNPs case, people imagine how much time and money they'll save on not using a lawyer).
I wouldn't expect that to ever change.
Who is they? And why did you make this a left/right issue?
"mad libs" is a game where you have a set of text with a bunch of blank spaces and then the group fills them in with words to come up with a funny end result.
Doesn’t that call into question the decision making of folks on HN rather than being a positive view of the product?
Seems like the killing blow here was claiming it can outright replace legal advice. Wonder how much that lie made compared to the settlement.
But yes, HN in general is a lot more empathetic towards AI than what the average consensus seems to be based on surveys this year.
It's interesting that many have expressed something similar in regards to the current LLMs, for programming for example: that even if their output isn't exactly ideal, they still lower the barrier of entry for trying to do certain things, like starting a project in Python from scratch in a stack that you aren't entirely familiar with yet.
HN is a fickle beast
I think about this all the time. If I didn’t have a conscience, I would be retired by now.
You mean you'd be like any other corporation or property manager or attorney? If you operate in the confines of the law that's all that matters. If you give normal people the same power to litigate as a billionaire, that's a feature, not a bug.
This is uh.. Yeah. This is what I meant by having no moral or ethical qualms.
There are things that I find immoral which are not illegal. I do not do those things, even though legally I could.
"This is uh.. Yeah." - can you clarify this remark please?
What are you on about?
I refuse to compromise my morals just because something is legal. That's it.
My morals did not come from some god, and they did not come from laws. No idea where you are getting religious zealotry from.
>"This is uh.. Yeah." - can you clarify this remark please?
Sure. I stopped myself from saying something rude to you.
It’s not about whether the explanation is rational. It’s just that if you really think the law is the last word on ethics and morality then you must be so self interested or ignorant or evil that you’re already not in a position to understand any explanation that’s offered.
As a simple example, should we teach children to not lie, or explain that the real error is to get caught telling a lie that didn’t even maximize their profit potential? Do you want corporations or individuals that shape society to be better moral actors than children, or worse?
If you don’t understand this kind of thing already, it must be willful.. retreating to the letter of the law in simple situations like this to plead moral and ethical agnosticism is such a transparent and weak position, rhetorically. Everyone knows that you just plan to profit from your willingness to engage in some kind of bad behavior.
Most of these cases wouldn't be defended at all otherwise.
Did they provide value to the user? Yes, nearly any situation in life involving money can be improved with a top lawyer on retainer, but that isn't always viable or economical
Do you get a free pass to do shitty things as long as you do some good things too?
I am totally onboard with the concept of the business, just not this particular implementation of it.
Definitely a valid point, but I don't feel comfortable applying the same idea to inanimate entities like corporations.
>Sometimes people get off entirely based on who they are (e.g. athletes, execs, etc).
On a personal level, I have never agreed with this.
https://english.stackexchange.com/questions/574974/etymology...
I invented smaller variants of deliveroo, airbnb and uber in my mind, around 2008, but I thought no, the only way to make any money would be to exploit people and break laws. honestly, what held me back was more the hassle of lawyers to make it all work. I didn't think I could stomach the effort.
To me it looks like they automated some boilerplate legal forms and marketted it as "AI" to capitalize on the hype.
The article states that they use ChatGPT.
Maybe it's morally or ethically wrong to prosecute them and take their belongings?
You don't, I guess. That's fine.
And that's why you're not a criminal. This is a criminal act and approved and implemented by criminals. Main thing is in the US it usually pays to do white collar crime as long as it's not too big or too embarassing to politicians.
It will no doubt improve, but somebody has to confirm that there are no errors in the output. If it has 1 error in 10,000 now, let's say it has a 3 log improvement, so now it's 1 in 1 million.
Would that be ok for legal or medical decisions? I don't think so. How about business decisions? Nope.
As long as AI is generating output with errors, it's going to have limited use.
To date, DoNotPay has resolved over 2 million cases and offers over 200 use cases on its website. Though DoNotPay has not disclosed its revenue, it charges $36 every two months. Given this, it can be estimated that DoNotPay is generating $54 million in annual revenue, assuming that all 250K users subscribe for 1 year."[1]
$193K seems like a pittance compared to the money they're making off of this.
It's weird to think that the FTC is right about the investigation, but somehow flubbed the penalty
If my dog bites somebody, that's on me. It should be no different with a company.
I don't have any special knowledge of this specific case, but it's important to note as a general principle that often the point of these fines is as the start of a process. It creates a formal legal record of actual damages and judgement, but the government doesn't see massive harm done yet nor think the business should be dead entirely. They want a modification of certain practices going forward, and the expectation is that the company will immediately comply and that's the end of it.
If instead the business simply paid the fine and flagrantly blew it off and did the exact same thing without so much as a fig leaf, round 2 would see the book thrown at them. Defiance of process and lawful orders is much easier to prove and has little to no wiggle room, regardless of the complexity that began an action originally. Same as an individual investigated for a crime who ends up with a section 1001 charge or other obstruction of justice and ends up in more trouble for that than the underlying cause of investigation.
So yes, not necessarily a huge fine. But if there weren't huge actual damages that seems appropriate too, so long as the behavior doesn't repeat (and everyone else in the industry is on notice now too).
> "what this robot lawyer can do is astonishingly similar—if not more—to what human lawyers do."
For basic stuff this is 95% of the end product
I expect the benefits of increased efficiency will be seen as temporarily zeroed inflation for legal services (prices actually going down? LOL) and a bunch of rents, forever (more or less, from the perspective of a human lifetime) to whichever one or two companies monopolize the relevant feature sets (see also: the situation with digital access to legal documents). Lawyers will be more productive but I expect comp will stay about the same.
And I think that as someone fairly pessimistic about the whole AI thing.
Wow!! That seems so simple, and literally a few weeks to do in today's ecosystem, now thoroughly testing make take a little more time, but wow, I wonder if it was evening attempting to do RAG.
There is no world in which allowing that to happen is a good idea.
Access to legal services to poor will change things. In short term, judicial system will be overwhelmed and are forced to adopt new and efficient procedures.
Can't they ask ChatGPT to write some objection and, at the end of the prompt, put "but make it look like it was written by a lawyer" and send that to the court to waive the fine?
Please read my comment as a joke. The title really sounds funny!
Lawyers at huge firms or companies automate the hell out of their legal actions against normal citizens and get things wrong all the time. But it’s okay if they do it because they’re part of the same cabal keeping the legal system afloat. Say what you want negatively about some dark patterns and marketing BS, they’re making legal things affordable to the every day person.
The fine seems fair for overhyped marketing claims, but I hope they keep going and improving.
If that gets automated, we are in AGI territory.
The second paragraph… hard to make a call on that one.
This is a good question. Is the point of paying the lawyer the piece of "intellectual output" (contract, specific advice, legal briefs, etc) or the fact that they stand behind it? Is a company using AI liable if the agents sell something for a pittance? Do the AI agents have intent and standing?
Of course this didn't stop them raising $10m from credulous investors in 2021.
It felt like a shaky premise at best as far back as I can remember. Even "standard" things often have many intricacies that a person might not know to say, and it may not let them know/ask them about it.
As an example, think of all the questions TurboTax et. al ask about taxes.
As an example I parked my car to drop my daughter off at a party and paid online - but mistyped the little number for the car park and ended up paying for 3 hours parking somewhere across the country.
Naturally the private car park tries to charge me 20x the parking fee as a “fine” - which they can whistle for frankly. But they sent varying letters that sound but don’t actually say “court” or “legal action” (things like “solicitors action prior to court”
I kept sending them the same answer they kept rejecting it
Then they actually sued me in county court. Oh wow I thought I better pay. And as a court judgement is really bad on credit record (one above bankruptcy) it’s serious. But I checked the court website anyway to be extra careful. And Incoukd challenge it - actually appear before the beak and say “hey it’s not my fault”.
So I filled in the form that says “yes I will challenge it, see you in court”
That might my wife said don’t be fucking stupid they have won pay them
So I went back the next day - and guess what - they had after 9 months withdrawn their action against me, no further need to progress, cancelled
I called the court to find out WTF
They had, and do every week, mass spammed the court with hundreds of parking cases, knowing that pretty much everyone would act like my wife pay a couple of hundred quid not to risk their credit record. I mean a county court judgement and you can kiss a mortgage goodbye.
That is simple abuse - an overworked courts system, hundreds probably thousands of rubbish claims that are put simply to strong arm people to pay up with legal threats, and no genuine attempt to filter out cases with merit, or even only look at “repeat offenders”
But is it worth the time of any parliamentarian to take this on (well frankly yes it would be great backbencher cause celeb, but what do I know.
Anyhow there was a point here - there are many many legitimate companies whose ficking business model is based on legal strong arming anyone who makes a minor infraction and that’s ok, but having a scam my business model to fight the scammy business model is bad?
Yes DoNotPay could have stayed on the right side of the line - but then would frankly run out of money. I guess we can only put our Hope in the hands of our elected representatives:-)
Especially when you are really just shoving data into an LLM and expecting a response to do some job, you are not training it to do a specific task.
Like the home buying AI that was on HN yesterday.
And yet people keep thinking so, both selling it as magic, and buying it as magic, and not once taking the time to consider what the words on the paper might mean.
For the most part, everyone can do these things, but it's nice to pay someome else to do it; especially in fields where experience gives expertise and hopefully wisdom. Also, it's handy to hire a licensed practicioner in fields where government requires licensure to sell services.
Some people are really woried about making big mistakes that are expensive to clean when plumbing or wiring or lawyering. It's a legitimate thing to consider.
> Some people are really woried about making big mistakes that are expensive to clean when plumbing or wiring or lawyering.
Even worse, if you lawyer wrong (or wire wrong), you might not be able to clean it up - you just do your time or perhaps die.
Law is some kind of magic, though. Consider the case where I want to cast the "Lawyer" protection spell.
If I chant "I am invoking my right to remain silent. I want to contact my attorney" then the police must stop questioning me and provide me one [1]. The spell worked.
If I chant "This is how I feel, if y’all think I did it, I know that I didn’t do it so why don’t you just give me a lawyer dog ’cause this is not what’s up" then my spell is not strong enough and the police can interrogate me as they see fit [2].
And then there's the time where a wizard had to interpret a comma [3].
[1] https://www.nedbarnett.com/do-i-have-a-right-to-an-attorney-...
[2] https://uproxx.com/culture/louisiana-supreme-court-suspect-l...
[3] https://www.loweringthebar.net/2017/03/the-oxford-comma-use-...
And, I imagine most people reading this think I'm either insane, ignorant or uneducated, or will attach some other adjectives to further alienate me. It's okay, I already feel like an alien.
I'm not looking for an argument. I don't need to convince anyone.
I was hoping for a more receptive audience. Oh well.
Or for a technical analogy, I'm capable of learning any programming language you can throw at me. However, a business who wants to hire someone is going to prefer someone with experience in that language's entire ecosystem. It's not enough to know the syntax. That's the easy part. The harder part is knowing which parts to reference at a given time, which modules experienced devs would choose to solve a specific problem, etc.
Well, same here. I'm wholly capable of reading and understanding the words of a contract or a summons or a lawsuit. What I don't know is the significance of specific phrases in those things, or what I'm allowed to use as evidence on my own behalf, or which issues I might raise that a judge would dismiss as something learned in the first semester of law school. And that's why I'd pay a lawyer to address legal issues for me.
Back in the day, anyone who touted AI generated works by default exclaimed that the work was not as good as a human. That changed now but was a valid statement back then.
Because the company chose to publish ads with misleading claims about the efficacy of their system.
Any non-trivial code generated by it takes more time to debug than just writing it from scratch.
Its "art" is abominably bad and repetitive.
Text generated by AI reads like corporate ad copy written by several committees of committees.
"Lovecraftian nightmare" best describes its video output.
AI voice generation sounds like soulless ripoffs of famous voice actors (the Attenborough clone is the worst) with misplaced stresses and an off-putting cadence derived from being completely unable to understand the broader context of the work it is narrating.
Its explanations on things are inferior to the first paragraph of any wikipedia article on the query topic.
A child with a pirated copy of FL-Studio can make more interesting music.
The wall being erected by AI customer service agents between a problem and an actual human who might be able to solve it is frustrating and useless.
On top of all of that the answers it confidently gives (almost always with no sources) are often extremely wrong.
Is there a secret AI product everyone is using that is actually good?
Edit: AI is however extremely good at rapidly creating an endless stream of barely-passable content designed to distract very cheaply so I expect its use by marketing and social media firms to continue its meteoric rise.
It's about how common the code is which you are asking the AI to generate and how much contextual clues it has to get the generation right.
Snark aside, this is literally a quote from their marketing: "what this robot lawyer can do is astonishingly similar—if not more—to what human lawyers do."
So to claim that they “exclaimed that the work was not as good as a human” is inaccurate.
The service it purported to offer is licensed. We could sit around and talk shit about the threshold to be licensed, the bar association, or licensing in general. But that's all distraction.
The point is the legal system has implemented a quality standard for providing certain services. There is a new thing providing the same service in novel way. Why wouldn't the legal system expect proof of quality?