Fooling Production-Grade Classification Systems with Adversarial Traffic Signs [pdf] | Hacker News Reader

Fooling Production-Grade Classification Systems with Adversarial Traffic Signs [pdf]

arxiv.org

62 pointsmpweiher21 comments

Just if someone is interested, here is the link to the "landing" page, with the metadata and abstract of the paper:

https://arxiv.org/abs/1907.00374

Makes me think that there is still a long way to go here. All of the changes they made to the signs leave them still easily readable by humans. One could see how reflections, weather and other environmental factors would distort a road sign more than this. Speed limit signs are possibly the hardest example though because it requires the system to recognise the exact number on it, rather than just its existence. I wonder how much harder it would be to get it to misclassify a stop sign?

The road sign has to be modified in the specific manner of the attack; not just any distortion will do.

The training set for the classifiers must not contain real-world traffic signs which are weathered, rusted, peeled, bent, smudged, and spray paint tagged.

Don't forget perforated by shotguns.

And stickered.

And those ones you sometimes find in malls, apartment complexes, or office parks that are an architectural standard sign with the traffic control sign symbol painted on one side, so you can't identify what the sign is from other angles. I hate those.

Wow I've read about adversarial images before but didn't think attacks would be this practical. It's weird to think that the added smudges have any relevance on the result when there is so much contrast. It appears easy to us because we're constantly training, cross-checking, and adjusting to conditions without much conscious interference.

At some point, not now, we will accept what the machine says. Machines will be better (faster and more accurate) at number recognition than us. But this paper shows how far behind they are still. In the context of cars I think the machines will just refuse to drive like we do. They (really their builders) will demand much better controlled conditions to operate safely.

We can still be fooled by sirnple things. Whats a siRNple?

Dealing with typos is the sort of robustness that people have, but that these image recognition programs are not displaying.

Nice! I really appreciate the real-world testing and not just "proving" the NN can be fooled by running it against image files.

This always seemed to me like a huge omission in reports about adversarial attacks, sure it can be fooled with pixel-perfect changes of input data, but would it survive literally any "real-world"-like environment? I am quite surprised it does!

https://m.xkcd.com/1958/

Not quite the same problem but the result is pretty much the same. The fact that computer vision can be fooled by attacks like these are lower risk than we might imagine. The more important question is what is the likelihood that such conditions occur naturally and randomly.

Edit: also, the summary comment about this being the first time a physical object was used to fool a commercial classification system sounds... dubious...

The XKCD you linked is reasonable on the surface of it, but if you mull over it a bit more, you'll see the three glaring omissions:

1) people care much, much less about "hurting" automated systems, and

2) people feel much less responsibility for accidents & failures if they are separated by a level of indirection of automation

3) people feel much less responsible for mass actions that affect anonymous people far away

Together, those three factors will indeed cause more incidents of people attempting to fool the self-driving cars. Especially if done by displaying malicious images via mass devices, like the light-up info boards, electronic billboards etc.. I don't expect much more actual accidents, tho.

Also, people aren't generally fooled by bad road lines.

Perhaps not. And they certainly aren’t fooled by adversarial speed signs. But they are fooled by plenty of other things which are equally easy to implement and equally dangerous.

While I don’t disagree with any of your points, there are still going to be people in cars, and any attempt to cause them to behave erratically could easily become attempted murder charges. I wouldn’t expect this to be any more common than people throwing rocks at cars on the highway.

This seems like a meaningless attack. If you’re modifying a sign, you can simply replace it with any other sign.

What exactly is the utility?

Yes, just think about a fleet of autonomous trucks carrying loads of valuable merchandise. A couple of these roadsigns that normal human cops would not detect, and you can redirect the entire fleet to your lair of doom.

That's assuming the vehicle doesn't have other means of locating itself, such as GPS.

An attacker could very well use a GPS jammer. Treating the navigation problem as a graph problem, placing these jammers where's needed in order to make the truck to follow the signs alone.

Your line of thinking is really good. A well-crafted attack is hard to execute and little things can very well ruin an attack sequence. A subtle the difference between academic research and the real world.

The threat aspect of this is, indeed, largely a red herring. The relevance here is the current lack of robustness in the face of ambiguity. The line between images that are properly identified and those that are not is very complicated, not well characterized, and (to some extent[1]) implementation-dependent. One might say that this is true of people too, but people have various higher-level skills for detecting and resolving ambiguity.

[1] As this study found adversarial images that affect a number of different implementations, there is some implementation-independence, which may be helpful in solving the problem.

The near-invisibility of the attack to a layman's (human) eye. I would think this kind of attack would take a non-negligible time to discover, unlike simply replacing a sign which is obvious to any random passerby.

Why would you think so? The signs that worked would look very odd when driven past. Tiny pictures on a screen don’t do these things justice.

On the other hand a different sign is just a different sign.

Reply on news.ycombinator.com