Adding some instances of these glasses to the training set would probably fix the model for these glasses, but then it might break for lipstick, hats, scarves, other glasses, or what-not.

The most interesting thing to me about this is that the models could be so badly fooled while obscuring such a small amount of the face. It shows the limits of solving problems by training zillion-parameter models to categorize zillion-dimensional data points: they won't necessarily be doing what you think they are, and may go completely off the rails when presented with data not close enough to their training sets.