An alternative way to think about it would be a padlock and a key. Anyone can close the padlock (assuming the common spring loaded design), but only the key holder can open it.
While this works for encryption, it doesn't really work for explaining signatures or however. Maybe someone can come up with a good analogy for that case.