Hacker News Reader

At Quobyte (https://www.quobyte.com) we use FUSE for the client for parallel file system access.

You can get dozens of GB/s out of FUSE nowadays. This will even improve in the near future as FUSE is adding io_uring support for communication with the kernel (instead of a pipe).

Yea I don't doubt that FUSE can go past NFS

But, as sibling is pointing out, e.g. Amazon fsx Lustre throughput is like 1GB/s throughout per TB storage, so presumably hundreds of GB/s at scale.

Yes there is AWS FSx with lustre in the blog..that might be worth checking out

This is a great idea

You could also try JuiceFS & Weka (if you can have access to a cluster).

A well configured and distributed Lustre will be very fast, BTW.

JuiceFS: https://juicefs.com/en/

Weka: https://www.weka.io/

Will test out Weka, Thanks for sharing