Hacker News Reader

Comment by klabb3 | Hacker News Reader

I’m surprised to see no mention of the attack vectors that – for lack of a better term – I call “drowning in crap”:

- send plausible phishing messages personalized to you (from data leaks, brokers) goes from expensive to cheap and automatable

- evade content based filtering such as spam filters to flood people with more crap

- generate so much plausible but regurgitated crap content for ranking systems (search, product reviews, friend requests)

- etc, etc

These things are already before LLMs one of the biggest problems across the board online. I would expect LLMs in particular to disrupt that power balance significantly in favor of the spammers and scammers. Is the implicit lack of attention to this issue that it’s not that bad? Won’t get worse? Or are we just supposed to learn how to swim in a sea of crap?

View on HN

As one of the posters below mentioned, this focuses on attacks on AI systems, not attacks by (or utilizing) AI systems.

In general there is a concerning apathy in the infosec community about the implications of generative AI on offensive tooling and what it means for already overwhelmed defenders. Seems to be much more of a focus on protecting against prompt injection, etc which doesn’t make much sense when you consider there are many, many models on HuggingFace that can easily be fine tuned to perform whatever nefarious task you may desire - no prompt injection required whatsoever.

We wrote up more thoughts here: https://www.wraithwatch.com/post/adapt-or-die-generative-ai-...

The link pertains to attacks on AI systems, not attacks using AI systems.

I assume that would be under "Poisoning"

In the spirit of endless hn pedantics… Poisoning implies a certain amount of finesse. All poisons of course being enzyme inhibitors in themselves and enzymes being a fancy biological term for a catalyst… but digress and no. I think drowning is very much the apt descriptor of this phenomenon.

"poison scales with dex, drowning is a weapon passive"

No, double checked:

> Poisoning attacks occur in the training phase by introducing corrupted data.

The distinction matters because poisoning (and all other listed attacks) impact the quality of the product and their users: I’d argue that AI providers are already incentivized to fix those. Whereas if say a hypothetical company ClosedAI is providing scammers/spammers with daily fresh loads of crap to use outside of the platform they simply make money from that. In other words, they have a first-order-incentive to be suppliers for these grey/bad actors. I guess this would be similar to the relationship between ad-platforms and scammy/shady advertisers.

during Trump's tenure, there was likely a whole bunch of bots betting on market movement and smearing the shit to make it happen.

maybe the political shit is fearful, but if capitalism requires a market and that market requires clear signals, well, here now is the greatshitstorm of our times.

Reply on news.ycombinator.com