I've tried to use virtio-pmem + DAX for the page cache to not be duplicated between the guest and the host. In practice the RAM overhead of virtio-pmem is unacceptable and it doesn't support discard operations at all. So yes a better solution would be needed.
I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.
Not really, if you develop inside containers in k8s then you will have a hard time:
- Installing something into the container
- Run docker/podman/k8s etc...
For me it's generally k8s. I'm currently migrating most of my stuff to a new setup :) I want to give headscale a shot. I've used my k8s to run Gitlab CI jobs and some of my apps/toy projects.
Can't you use NixOS for this stuff? I think it's the main selling point of NixOS that every developer has the same environment. You may also deploy those nix environments via terraform.