I assume it's pretty slow, not on storage but on compute. x86 docker/podman on M1 must use QEMU software emulation, rosetta doesn't support x86 virtualization.
BTW, in case you don't want to depend on a fork, upstream podman is going to gain M1 support (in the sense of 'podman-machine' knowing how to start aarch64 VMs with hvf) very soon.