Comment by 015a | Hacker News Reader

Comment by 015a | Hacker News Reader

> “I also think whoever was responsible for it should be fired,” she added.

> “We were made aware that a single user of the AI platform was purportedly provided an image that did not adhere to the built-in parameters set by the developers. Prior to launch, we agreed to a comprehensive set of rules to govern image creation, including that people in images be fully clothed. We have not seen the image that the AI provided to this user, but out of an abundance of caution we took the website down,” the spokesperson confirmed to the Jason Rantz Show on KTTH.

Here's a tangential take to all this, but its the overwhelming thing I feel reading this: Something happened to our society which resulted in effectively no one being able to be held accountable for anything. Its always "well, we wrote this policy and we were well within the policy! the thing just went oopsie daisy no one could have predicted that!"

They were creating and distributing porn of non-consenting individuals. Everyone involved with this should be fired, and maybe actually it should go further than that.

I feel this tremendously in the day-to-day of the tech world. I will make some system change which brings down production. I hop on a postmortem with my boss, and the first thing she says: "We have a blameless engineering culture. Its not your fault, and I don't hold you responsible." I disagree, entirely: it is my fault. I need to own it, and I don't understand how people function tip-toeing around accepting responsibility for something that is obviously their fault. How do you, the individual, improve?

And the line for your responsibility is where exactly? You want to be personally responsible for every single dependency you pull in, every piece of hardware between you and the end user? This is a farcial suggestion.

> They were creating and distributing porn

Unless they specifically trained their own model to do this - which they obviously didn't - this is just a bug. If we all got fired any time a bug happened, society would grind to a standstill. Shit happens, move on. What's important are intentions and taking reasonable precautions.

> You want to be personally responsible for every single dependency you pull in

Yes.

> every piece of hardware between you and the end user

Nah. See how you said "where do you draw the line" and then proceed to just ignore drawing the line? I'm ok with drawing a line. There doesn't have to be logic or reason behind it. That's where I draw it; the repositories and websites and apps that I distribute.

> Unless they specifically trained their own model to do this - which they obviously didn't - this is just a bug.

Go try to make Midjourney generate a nude image of someone. Its extremely difficult. You'd have to be extremely intentional and, frankly, malicious. I've never seen it happen.

The way this article is written, it sounds like the user just uploaded a photo and it happened. So, what kind of prompting are they using which led to this? Also: They said they don't have a copy of the image, but that's extremely hard to believe just assuming how systems like this would be designed; the image has to be served from somewhere, are they seriously purging the images from e.g. their storage buckets that quickly? Possible, but hard to believe; of course, they'd say "oh we don't have a copy" if they investigated, found it doing this, and just wanted the story to die.

Or, we could do as you want, and just sweep this under the rug. Nothing to learn. No one to blame. Just a quirky little thing that happened! So quirky! A state gambling office manufacturing pornography of random people to sell lottery tickets! Quirky and fun!

Well it looks like they've drawn a line too and found themselves not responsible, so they're already doing what you're advocating it seems. Nothing to learn here.

> They said they don't have a copy of the image, but that's extremely hard to believe just assuming how systems like this would be designed; the image has to be served from somewhere, are they seriously purging the images from e.g. their storage buckets that quickly? Possible, but hard to believe; of course, they'd say "oh we don't have a copy" if they investigated, found it doing this, and just wanted the story to die.

I don't find it very hard to believe.

1. If the image can be generated in a few seconds, the easiest way to implement this is to accept the file upload in a request, do the AI stuff, and return a new image in the response. In this case, it would be extra (and unnecessary) work to save it in a storage bucket. Both the uploaded and generated images only exist in memory for the lifecycle of the http request, and then get deallocated.

2. If the image _is_ being saved in a storage bucket (or Redis, or whatever), the bucket might have a lifecycle policy on it that automatically expires images after a set time. This is a pretty common cost-saving measure, so it's not too unlikely.

A decision maker figured they could cut costs by using a DIY model instead of paying for mid journey API calls. They're the ones who are responsible.

But I wouldn't trust any diffusion image gen here. They're all liable to create disturbing and inappropriate imagery, and there's no way to prevent it.

If you were messing around with a gun that had the safety on, and the gun malfunctioned and shot a child while you were waving it around, would it be a valid defense to say that you took all the necessary precautions (since the safety was on, after all) and it's just a silly oopsie that the gun did that?

You should have a responsibility to put the necessary safeguards on a potentially dangerous tool that you're using and to do your due-diligence if you're using that tool. It would have been impossible to accidentally produce porn of a non-consenting individual if they had not been using an AI image manipulation pipeline at all. If you do choose to use an AI image manipulation pipeline, you should take on the responsibility to make sure that it's not going to do illegal things you didn't tell it to do. If there's a risk of causing harm with it, just don't use it! You don't need to be producing cute images of people in Vegas or whatever, and if doing so necessarily implies a risk of accidentally creating porn of them instead, you could (and in fact, have a responsibility to) just choose to do neither.

A gun malfunction has a potentially lethal consequence. This bug caused a person to see a fake photo of herself with fake boobs out. Perhaps it's ok that the safety standards for the latter are lighter, no?

You also didn't address the main point of the earlier comment. Is every programmer responsible - even criminally, some suggested - for potential bugs or vulnerabilities in one of their products?

Did they actually distribute the image to anyone else other than the woman? The developers say they hadn't seen the image nor was the journalist able to find it.

Sure, the developers said they hadn't seen the image and that they'd never seen the image generator do this and etc etc. That's regardless of the truth; there's no reality where they'd say anything else, and their word cannot be trusted. That's the point of an investigation. They have every incentive to make this disappear as quickly as possible.

And, to be fair, its doubtful it would get very far; there's no evidence the aggrieved party wants to sue, she doesn't have the photo it generated, etc. I'm not standing on a soapbox arguing that the secret service should get involved or something. I'm just asserting that, it seems to me, we the people should care a bit more about this than we seem to be. Other people saying "its just a bug!" are incomprehensible to me; there's a spectrum between "your signin button is using the wrong hex code" and "your chemotherapy dispenser is outputting the wrong volume of medicine", and it is not obvious to me that this bug isn't closer to the right side of that spectrum than left.

An engineer in the room might even know that these generative systems are capable of hallucinogenic, and sometimes even explicit, output; but they don't have the political capital to stop decision makers from deploying these systems, or underinvesting in safeguards, etc. That's why we need pushback, ideally lawsuits but at least public outcry; so decision makers treat the decision to use generative AI with the care and consideration it deserves.

Reply on news.ycombinator.com