The filesystem performance degradation was resolved in kernel 5.11 which added support for rootless overlayfs.

The network performance is caused by slirp (usermode TCP/IP) but it is being resolved too : https://github.com/rootless-containers/bypass4netns