Believe it or not, if you want to invest in gold, the only option isn't buying and storing physical gold in your home. That might even be a dangerous idea.
Stocks are certificates which claim partial ownership of a company. But how can owning a piece of paper be like owning a part of a company? What is a company anyway but an abstraction over the concept of group liabilities, assets, contracts, and actions? Or do you think companies only exist to allow people to break laws under another identity?