It's an annoying that you can only make snapshots on a stopped container. With VMs it works in a running VM.
But even if they had, the RAM snapshot needs to be written, but without freezing the container. I would appreciate an option when I could ignore everything that was not fsyned, e.g. Postgres use case. In that case the normal ZFS snapshot should be enough.
The state of a container is not part of the snapshot (just checked), as it is really hard to capture the state of the container (CPU, network, all kinds of file and socket handles) and restore it because all an LXC container is, is local processes in their separate cgroups. This is also the reason why a live migration is not really possible right now, as all that would need to be cut out from the current host machine and restore in the target machine.
This is much easier for VMs as Qemu offers a nice abstraction layer.
We've pretty much stopped using LXC containers in Proxmox because of all the little issues.
Things that got me recently, just off the top of my head:
- htop sees memory usage incorrectly
- Docker tries to use overlay2 on ZFS which fails (I think, I needed to create and mount an ext4 volume for reasons)
- Hashicorp Vault needed disable_mlock because I believe LXC blocks the syscall.
On the other hand, I like my Samba file server in a container though, because it is much easier to share storage from the host into LXC than a VM.
LXC and VMs both have pros and cons.