What am I, the average Joe, going to do with a few ounces of silver though? A few scraps of paper or a handful of metal, both seem pretty equally useless.

Food, medicine, clothes, transportation... Now those are far more inherently useful to a human. Not great mediums for a standard of exchange, however.