Yep! But I don't have access to any block devices on nodes, only the local paths so I'm not sure OpenEBS or Ceph would work...
Have you thought about TCP sockets between the apps and sharing state, or something like a redis database?
In this example, I have 200GB of ephemeral storage available on each node, ideally I'd like something like this:
node1: /tmp/data1 (200GB free space)
node2: /tmp/data2 (200GB free space)
node3: /tmp/data3 (200GB free space)
node4: /tmp/data4 (200GB free space)
node5: /tmp/data5 (200GB free space)
...pods could some how mount node{1..5} as a volume, which would have 5 * 200GB ~1TB of space to write to... multiple pods could mount it and read the same data.