It needs to be either entirely precious, entirely worthless (and buried), or exceptionally large. Nobody has stolen the pyramids yet.
What I would think is that some, if not all, of the ordinary stones will indeed by lost, but then surreptitiously replaced by others (after all, they have no identifiable properties).
1300 years from now, there will be stones, but the not the original ones. I think it fits the purpose perfectly, just like purported official Christian relics.
While not ideal, even that may be OK to some extent -- perhaps the valuable thing is the soul of what you are carrying. I think of it like a University or a specific house of worship. Yes, there are physical assets, but the real assets (the people) come, contribute, leave but the soul of the organization lasts for hundreds of years.