Introduction to Adversarial Machine Learning

What I would like to understand better is how transferable these adversarial images are from a network to another. I mean is the adversity "deep", like practically all models built on resnet18 (or even completely different models) suffer from the adveristy? Or is it "shallow", meaning just one more training round will mess up the special "gradient paths" found for these particular network weights?

> Not worried enough? Imagine an attacker who manipulates road signs in a way such that self-driving cars will break traffic rules.

If someone was manipulating a bunch of road signs for bad, human drivers would probably be in trouble too..I guess the point is that these attacks could be more subtle/scalable?

This confirms my opinion that the only way for self driving cars to work is for every car on the road to be a self-driving car. Stop signs would become like train semaphores where rather than being purely visual, there are other electronic indicators that tell the car to stop at a certain point. The roads should then be more like a train network. I really think this will mean more efficient use of space since you can have shorter following distance (this was achieved on the train network too once automated systems took over from human drivers). Trying to blend self-driving and human driven cars is just going to cause a huge mess. Accounting for human mistakes and AI mistakes is just too big a problem for AI to solve right now, and imo, for a very long time.

I find adversarial attack super interesting. It opens up a field in machine learning where people questions the robustness of models. Adversarial examples exist because of many reasons, with data and weight space being high-dimensional as one.

IMO, the world needs to pay more attention to this, as we implement more and more AI applications. What worries me more is that we still don't really have a good solution to his problem. Pretty much every model that is defended can be attacked in another way.

My guess for now is that we would have to go all the way back, and question about the assumptions we are making about high dimensional spaces, and use a more rigorous way to prove theorems and lemma about AI systems, rather than experimentally saying the accuracy is high enough therefore it works.

It's hard for me to imagine how an adversarial machine learning attack would work in practice. Sure, you might be able to fool Shazam, but it's not like Tesla or Waymo are going to give you free access to their models.

Is this a sufficient risk that an AI company would be wise to hire an adversarial security engineer? Or is this still well below hypothetical?