People frequently don't read signs, which is the point of the affordance design (plus caters to folks who can't read the given language). Also, hands make stuff dirty - you do want something there that's easy to clean, though that can just be a metal plate that can't be pulled.