Ok ... So that makes a little more sense. It really struck me that I just didn't know how to
trust the persistence. In a world where the application could run on any virtual cluster and even across different sets of physical hardware. (Looking at CoreOS for example as a system designed to appear as "one machine" despite spanning multiple hardware instances).
So when I specify the VOLUME (in a docker file/kubernetes-pod-spec) this is managed by the system.
And with respect to ensuring uninterupted service on the database layer then, how are ongoing changes synched? Or is that something I must design for by adding a queueing layer?
To make that concrete, I have one pod with MySQL and the associated VOLUME is configured on KUBERNETES instance.
I tell K to scale MySQL to 2 instances.
It points the second instance to the same physical files and starts up?
So VOLUMES live outside of KUBERNETES and do not spin up or down in a transient nature?
Can 2 instances of a database server use the same files without stepping on each other?
How does the VOLUME scale without becoming the new single point of failure?
Maybe that is the point where you use massive RAID striping and replication to scale storage? (That's far from a new concept and is a pretty stable tech).
If I'm on the right track here I might almost be ready to say I understand and trust persistence!