If you don't pay your debts or taxes then even items that you own and which are in your possession can be taken from you, thanks to the rule of law. Conversely, if someone comes along and takes it from you without legal justification, the rule of law allows you to claim it back. That same rule of law ensures that you are entitled to the gold you own that isn't in your possession.

Drawing the line at possession is arbitrary. Either way your ownership is governed by law, and is only as good as the strength of that law.