Using a hardware wallet doesn't mean you're physically storing it. It's just a user interface to make storing it a little easier and a little more secure than having it as a file on your computer or on a piece of paper.

I'd recommend keeping it at the COMEX gold warehouse.