An intro to DeepSeek's distributed file system
maknee.github.io
maknee.github.io
What's important here with S3FS is that it supports (1) a fuse client - it just makes life so much easiter - and (2) NVMe storage - so that training pipelines aren't Disk I/O bound (you can't always split files small enough and parallel reading/writing enough to a S3 object store).
Disclaimer: i worked on HopsFS. HopsFS adds tiered storage - NVMe for recent data and S3 for archival.
[ref]: https://www.hopsworks.ai/post/scalable-metadata-the-new-bree...
And NVMe optimisations e.g. NVMeoF in OpenEBS (Mayastor).
None of it is particularly ground breaking just a lot of pieces brought together.
The truly amazing part for me is combining NVMe SSD + RDMA + supports reading a huge batch of random offsets from a few already opened huge files efficiently. This is how you get your training boxes consuming 20~30GiB/s (and roughly 4 million IOPS).
Tectonic is built on ZippyDB which is a distributed DB built on RocksDB.
>What's important here with S3FS is that it supports (1) a fuse client - it just makes life so much easier
Tectonic also has a FUSE client built for GenAI workloads on clusters backed by 100% NVMe storage.
https://engineering.fb.com/2024/03/12/data-center-engineerin...
Personally what stands out to me for 3FS isn't just that it has a FUSE client, but that they made it more of a hybrid of FUSE client and native IO path. You open the file just like normal but once you have a fd you use their native library to do the actual IO. You still need to adapt whatever AI training code to use 3FS natively if you want to avoid FUSE overhead, but now you use your FUSE client for all the metadata operations that the native client would have needed to implement.
https://github.com/deepseek-ai/3FS/blob/ee9a5cee0a85c64f4797...
From a theoretical point of view, like others have pointed out, parallel distributed file systems have existed for years -- most notably Lustre. These file systems should be capable of scaling out their storage and throughput to, effectively, infinity -- if you add enough nodes.
Then you start to ask, well how much storage and throughput can I get with a node that has X TiB of disk -- starting to evaluate efficiency. I ran some calculations (against FSx for Lustre, since I'm an AWS guy) -- and it appears that you can run 3FS in AWS for about 12-30% cheaper depending on the replication factors that you choose against FSxL (which is good, but not great considering that you're now managing the cluster yourself).
Then, the third thing you start to ask is anecdotally, are people able to actually configure these file systems into the size of deployment that I want (which is where you hear things like "oh it's hard to get Ceph to 1 TiB/s") -- and that remains to be seen from something like 3FS.
Ultimately, I obviously believe that storage and data are really important keys to how these AI companies operate -- so it makes sense that DeepSeek would build something like this in-house to get the properties that they're looking for. My hope is that we, at Archil, can find a better set of defaults that work for most people without needing to manage a giant cluster or even worry about how things are replicated.
SSDs Have Become Ridiculously Fast, Except in the Cloud: https://news.ycombinator.com/item?id=39443679
I worked this out to the macOS SMB implementation really sucking. I set up a NFS driver and it got about twice as fast but it’s annoying to mount and use, and still far from the disk’s capabilities.
I’ve mostly resorted to abandoning the network (after large expense) and using Thunderbolt and physical transport of the drives.
I mount my nfs shares like this: sudo mount -t nfs -o nolocks -o resvport 192.168.1.1:/tank/data /mnt/data
-o nolocks Disables file locking on the mounted share. Useful if the NFS server or client does not support locking, or if there are issues with lock daemons. On macOS, this is often necessary because lockd can be flaky.
-o resvport Tells the NFS client to use a reserved port (<1024) for the connection. Some NFS servers (like some Linux configurations or *BSDs with stricter security) only accept requests from clients using reserved ports (for authentication purposes).
NVMe i bought for 150 dollars with 4 TBs capacity gives me 6000 MB/s sustained
https://docs.aws.amazon.com/ec2/latest/instancetypes/so.html
> NVMe i bought for 150 dollars
Sure, now cost out the rest of the server, the racks, the colocation space for racks, power, multiple AZ redundancy, a clos network fabric, network peering, the spare hardware for failures, off site backups, supply chain management, a team of engineers to design the system, a team of staff to physically rack new hardware and unrack it, a team of engineers to manage the network, on call rotations for all those teams.
Sure the NVME is just $150 bro.
For example 3FS looks like its optimised for read throughput (which makes sense, like most training workloads its read heavy.) write operations look very heavy.
Can you scale the metadata server, what are the cost of metadata operations? Is there a throttling mechanism to stop a single client sucking all of the metadata server's IO? Does it support locking? Is it a COW filesystem?
SeaweedFS is more about amazing small object read performance because you effectively have no metadata to query to read an object. You just distribute volume id, file id (+cookie) to clients.
3FS is less extreme in this, supports actual POSIX interface, and isn't particularly good at how fast you can open() files. On the other hand, it shards files into smaller (e.g. 512KiB) chunks, demands RDMA NICs and makes reading randomly from large files scary fast [0]. If your dataset is immutable you can emulate what SeaweedFS does, but if it isn't then SeaweedFS is better.
[0] By scary fast I mean being able to completely saturate 12 PCIe Gen 4 NVMe SSD at 4K random reads on a single storage server and you can horizontally scale that.
With these systems, the runtime guarantees of data integrity are very high and the failure rate is very low. And best of all, failure is constantly happening as a normal activity in the system.
So once you have data integrity guarantees that are better in you runtime system than your backup process, why backup?
There are still reasons, but they become more specific to the data being stored and less important as a general datastore feature.
Because of mistakes and malicious actors...
Redundancy and backups are not the same thing! There's some overlap, but treating them as interchangeable will occasionally result in terrible outcomes, like when a config change that results in all 5/5 datacenters fragmenting and failing to create a quorum, then finding out your services have circular dependencies when you are trying to bootstrap foundational services. Local backups would solve this, each DC would load last known good config, but rebuilding consensus necessary for redundancy requires coordination from now-unreachable hosts.
Remember that there are two purposes for backup. One is hardware failures, the second is fat fingers. Hardware failures are dealt with by redundancy which always involves keeping redundant information across multiple failure domains. Those domains can be as small as a cache line or as big as a data center. These failures can be dealt with transparently and automagically in modern file systems.
With fat fingers, the failure domain has no natural boundaries other than time. As such, snapshots kept in the file system are the best choice, especially if you have a copy-on-write that can keep snapshots with very little overhead.
There is also the special case of adversarial fat fingering which appears in ransomware. The answer is snapshots, but the core problem is timely detection since otherwise you may not have a single point in time to recover from.
Non-local storage is definitely worthwhile there. Snapshots can have additive value there because they may be more coherent states.
Speaking from experience working at a hyperscaler - 1. cross-regional mirroring 2. Good old tape backups
Because of the replication factor here, I assume that this filesystem is optimised for read throughput rather than capacity. Either way, there is a concept of "nearline" storage. Its a storage tier that is designed to be only really accesed by a backupagent. The general idea is that it stores a snapshot of the main file system every n hours.
After that you have as many snapshots as you can afford.
Presumably the key necessary features are PB's worth of storage, read/write parallelism (can be achieved by splitting a 1PB file into say 10,000 100GB shards, and then having each client only read the necessary shards), and redundancy
Consistency is hard to achieve and seems to have no use here - your programmers can manage to make sure different processes are writing to different filenames.
Famous last words.
It is very common when operating data platforms like this at this scale to lose a lot of nodes over time especially in the cloud. So having a robust consistency/replication mechanism is vital to making sure your training job doesn't need to be restarted just because the block it needs isn't on the particular node.
But the type of consistency they were talking about is strong ordering - the type of thing you might want on a database with lots of people reading and writing tiny bits of data, potentially the same bits of data, and you need to make sure a users writes are rejected if impossible to fulfil, and reads never return an impossible intermediate state. That isn't needed for machine learning.
What follows is a long period of saying "see, distributed systems are easy for genius developers like me"
The last words are typically "oh shit", shortly followed oxymoronically by "bye! gotta go"
I think their "Other distributed filesystem" section does not answer this question.
If my systems guys are telling me the truth is it a real time sink to run and can require an awful lot of babysitting at times.
Among other things, the OSD was not designed with NVMe drives in mind - which is fair, given how old it is - so it's nowhere close to being able to handle modern NVMe IO throughput and IOPS.
For that you need zero-copy, RDMA etc.
Note that there is a next-generation OSD project called Crimson [0], however it's been a while, and I'm not sure how well it's going. It's based on the awesome Seastar framework [1], backing ScyllaDB.
Achieving such performance would also require many changes to the client (RDMA, etc).
Something like Weka [2] has a much better design for this kind of performance.
I do agree that nvme-of is the next hurdle for ceph performance.
I have 3x Samsung NVMe (something enterprise w/ PLP; I forget the model number) across 3 nodes, linked with an Infiniband mesh network. IIRC when I benchmarked it, I could get somewhere around 2000 MBps, bottlenecked by single-core CPU performance. Fast enough for homelab needs.
I benchmarked 1 vs 2 OSD and found 2 OSD was better. I don’t think it is recommended to run more than 2 OSD per NVME.
"We were reading data at 635 GiB/s. We broke 15 million 4k random read IOPS."
Source: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/
I don't know man, I think 15M random read IOPS is actually quite fast. I've built multi million IOPS clusters in enterprise settings all on nvme in the past.
680x NVMe SSDs over 68 storage servers (so 68 CPUs) for just 15M (or 25M, tuned) random read IOPS is pretty underwhelming. The use cases where 3FS (or some other custom designs) shine are more like, 200M random read IOPS with 64 servers each with 8 PCIe gen 4 NVMe SSDs (512x SSDs in total).
It is a complete bear to manage and tune at scale. And DO never greenlit offering anything based on CephFS either because it was going to be a whole other host of things to manage.
Then of course you have to fight with the maintainers (Red Hat devs) to get any improvements contributed, assuming you even have team members with the requisite C++ expertise.
Minio is absolutely not datacenter-scale and I would not expect anything in Go to really reach that point. Garbage collection is a rough thing at such enormous scale.
I bet we'll get one in Rust eventually. Maybe from Oxide computer company? Though despite doing so much OSS, they seem to be focused around their specific server rack OS, not general-purpose solutions
Crucible is our storage service: https://github.com/oxidecomputer/crucible
RFD 60, linked in the README, contains a bit of info about Ceph, which we did evaluate: https://rfd.shared.oxide.computer/rfd/0060
In particular for my homelab setup I'm planning to run JuiceFS on top of S3 Garage. I know garage is only replication without any erasure coding or sharding so it's not really comparable but I don't need all that and it looked at lot simpler to set up to me.
For the most part I can live with the latency. For frequently or repeatedly accessed files I believe JuiceFS also has a caching layer to bring down the latency on those. My use cases are quite different from what DeepSeek requires. I was just curious whether I needed to investigate 3FS since it's the new kid on the block.
What appeals to me about JuiceFS is that I can easily mount it into my containers for a distributed FS on a small cluster for some resilience. It looked simpler than Ceph or GlusterFS for that and it's backed by S3 so that I could swap out my S3 Garage setup for a cloud based S3 backend if required.
Edit: I am a DeepSeek newbie BTW, so if this question makes no sense at all, that's why ;)
In your example it is likely the US college has been authorized to use a DeepSeek model for research, not the DeepSeek 3FS distributed file system.
Understanding Smallpond and 3FS - https://news.ycombinator.com/item?id=43232410 - March 2025 (47 comments)
Smallpond – A lightweight data processing framework built on DuckDB and 3FS - https://news.ycombinator.com/item?id=43200793 - Feb 2025 (73 comments)
Fire-Flyer File System (3FS) - https://news.ycombinator.com/item?id=43200572 - Feb 2025 (101 comments)
Instead of spending ludicrous amounts on Nvidia GPUs, can just deploy commodity clusters of AMD EPYC servers with tons of memory, NVMe disks, and 40G or 100G networking which is vastly less expensive.
Goodbye Nvidia moat!
In principle you could use fiberchannel to connect a really large number (2²⁴ iirc) of disks to a single server and then create a single ZFS pool using all of them. This lets you scale the _storage_ as high as you want.
But that still limits you to however many requests per second that your single server can handle. You can scale that pretty high too, but probably not by a factor of 2²⁴.
Thinking about security risks is an odd concern to have?
Great security teams get paid to "worry", by which I mean "make a plan" for a given attack tree.
Your use of "odd" looks like a rhetorical technique to downplay legitimate security considerations. From my point of view, you are casually waving away an essential area of security. See:
https://www.paloaltonetworks.com/cyberpedia/data-exfiltratio...
> but you should not be exposing these kinds of services to the public internet (or giving them access to the public internet), eliminating the concern that it's giving your data away.
Let's evaluate the following claim: "a "properly" isolated system would, in theory, have no risk of data exfiltration. Now we have to define what we mean. Isolated from the network? Running in a sandbox? What happens when that system can interact with a compromised system? What happens when people mess up?
From a security POV, any software could have some component that is malicious, compromised, or backdoored. It might be in the project itself or in the supply chain. And so on.
Defense in depth matters. None of the above concerns are "odd". A good security team is going to factor these in.
P.S. If you mean "low probability" then just say that.
I think these kinds of concerns are more valid for storage systems which serve online traffic or have some kind of connection to the outside world.
Really? More rhetoric that minimizes legitimate security concerns?
Again, if someone wants to make the claim that such concerns are "low probability" that would at least be defensible.
https://www.cisecurity.org/insights/blog/deepseek-a-new-play...
> just like an asteroid hitting Earth
So much one-sided rhetoric.
Let's rephrase this. It is not simply that DeepSeek is a Chinese company. It is because of its links to the CCP [1] [2] and the CCP's cyber operations.
[1]: https://www.scmp.com/news/china/politics/article/3306943/chi...
[2]: https://www.journalofdemocracy.org/online-exclusive/why-deep...
I wrote a longer comment about this here: https://news.ycombinator.com/item?id=43745423
The idea that "you should not be exposing these kinds of services to the public internet (or giving them access to the public internet)" is naive. Aside from the fact that this requires every user to show that level of diligence - a completely unrealistic expectation - if a system is backdoored, it can also have means of exfiltrating data beyond just sending it out on the public internet.
And then there's the obvious point that if you suspect a device is compromised, it's completely irresponsible to use it anyway and assume that you're going to be able to prevent unauthorized access.
As some examples, there are documented concerns about Huawei/ZTE routers, which have been banned in Australia, New Zealand, Japan, Taiwan and the US for that reason. Unauthorized third-party code was found in Juniper Networks firewalls. Fortinet had hardcoded admin credentials in its firewalls and VPNs - probably a self-inflicted mistake, but still useful to attackers. Similarly, Western Digital NAS devices had a hardcoded backdoor account. D-Link routers had such a backdoor in their device web interface. There are many more examples like this.
Snowden revealed some of the US government activities in this area. The US, Russia, China, North Korea and other countries have all been involved in attacks involving BIOS/UEFI firmware, router firmware, NAS, and manufacturing supply chains. Covert exfiltration has been involved in many of these cases, using techniques other than transmitting over the internet.
And of course there was the recent (reported late 2024 Salt Typhoon attack by China on US and other Western telecom networks, which relied on these kinds of techniques, and gained access to large amounts of data, including audio and text of targeted people.