Hacker News Reader

Comment by cantSpellSober | Hacker News Reader

> the AI superimposes your image into the vacation spot

> She was shocked at one of the AI photos Washington’s Lottery spit out

For discussion: is it possible this was this a joke (not on the victim's part) in very poor taste?

Nah, it sounds like a simple latent space adjacency problem. When you give an image generation model the "swimming" keyword, that makes it much more likely to produce bathing suits and bikinis, and once it is already generating that much bare skin it is likely to keep going and make some bare breasts (if many images of bare breasts are trained into the model). Their claimed prompt of "swimming with the sharks" also introduces an element of danger and excitement, which some models could also use as a signal to introduce risqué elements. Basically, if models produced only what you asked for they would produce extraordinarily boring results. Many models come up with additional elements to add to the image based on the adjacent elements encoded in the latent space branching off of your prompt.

The main issue here would be their prompt choice ("swimming with the sharks" was a slightly risky prompt. It could have easily gone another way and generated a gory Jaws-inspired image of her getting eaten by a shark). Secondary as an issue is the model that they used (likely a model that has too many NSFW images trained in). The big lesson for corporate entities is to pay attention to the training data that was used in the model (if you want purely SFW results you have to use a model that has nothing NSFW trained in) and also carefully consider what imagery your keywords is likely to produce as a side effect.

Base model choice is a big one. Probably didn’t pay close attention to their negative keywords either. You can specifically tell models to strongly avoid making nsfw images.

Plus most image generation pipelines have an optional secondary step that checks for nsfw images and blocks the output before it reaches the user. They may have neglected that step.

Something nsfw could still have gotten through, but it’s unlikely given the many safeguards that most pipelines have built in.

Yep very true. My guess would be that the feature was implemented by someone very new to this whole AI generated image thing. They had fun implementing it, but did not do a ton of deep research into how to implement it safely, with negative prompts and follow up checks. This is one of the biggest challenges with generative model use. They are powerful, but easy to mess up.

Negative prompts are not perfect, especially for preventing nudity.

Several of the image generators will create pornographic images due to their training data. This really isn't unexpected if the devs weren't careful with the generation service chosen?

Nothing in this article screams 'careful'

Reply on news.ycombinator.com