That's correct. I wouldn't think of it as a VM (a container) though but rather as a server which happens to be virtual. Yes, that's literally just a different word for the same thing but the different emphasis affects thought patterns. For all intents and purposes, from the buyer's perspective, a VPS is a small server, not a different type of thing.

It's true you shouldn't put super sensitive data on a VPS because the host could access it. Regular sensitive is fine - your host will be in a world of trouble if they access your data without permission, so you can generally trust them not to read your emails or open your synced nudes. But if your data is so sensitive that the host would risk everything to read it, or would avoid getting in trouble at all (e.g. national security stuff) then absolutely don't use a VPS. For that level of paranoia you'd need at least a dedicated server which makes it unlikely the host has a live backdoor into the system, ideally your own server so you know they don't, and for super duper stuper paranoid situations, one with a chassis intrusion switch linked to a bag of thermite (that's a real thing).