You're misusing the term affordance (not blaming you, Don Norman himself has written about how it's constantly misused).

You can wave your hand over anything (unless something is blocking your way or is too far, etc.), therefore most stuff affords being waved over.

A mug, with or without a handle, affords "being held with your hand"(although more or less comfortably).

What you're talking about is a signifier.

Quoting from a random article:

As Don Norman explains it: “[…] although affordances make sense for interaction with physical objects, they are confusing when dealing with virtual ones. As a result, affordances have created much confusion in the world of design. Affordances define what actions are possible. Signifiers specify how people discover those possibilities: signifiers are signs, perceptible signals of what can be done. Signifiers are of far more importance to designers than are affordances.” [Donald A. Norman, The Design of Everyday Things: Revised and Expanded Edition, 2013, pXIV-XV]