It does because the coin has the private key. Maybe it's better to think of it as a key to a (public) safe. The key represents the right to open this safe. So, by giving you the key, I give you the ownership to the content of the safe. (However, if I have another copy of the key it get's problematic.)
> It's essentially an anonymous IOU
Which is exactly the same as normal cash. The only difference is that you can destroy them without really loosing the money. So you have a worthless piece of metal. You can do the same with your normals coins (eg. melt them down), but then the worth is just gone ;)