Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system.

Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.

You have to do that anyway for serious batch processing.

You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.

The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.

There are tools that enable this (e.g. look at temporal.io).

As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines.

In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.

That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done.

I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.