Show HN: Download HuggingFace Models/Datasets easily and super fast
github.com
Multithreaded downloading of large files (LFS)
Filter Downloads, specic LFS models files can be specified for downloading (Usefull for GGMLs)
SHA256 checksum
github.com
I cloned a repo bare from hugging face with git, and then I used `git lfs fetch`.
This first download using just git and git lfs was served to me at an average speed of 100 MiB/s. That is extremely fast, and I was surprised given that I was downloading about 30 GiB of data from them at this speed for the data of that first repo.
The second and third clones I made from two other repositories for also many many GiB of data were slower but still plenty fast. Both of these were downloaded at about 30 MiB/s average. Which is still crazy fast.
If downloads go slowly then it might be intentional because hugging face might need to slow down to serve everyone.
I think it is better to stick to git and git lfs for downloading from hugging face. They are serving us hundreds of gigabyte of data each, completely free of charge.
What I would like to see though is a tool that integrates git, git lfs and BitTorrent natively, so that I could git clone from hugging face and then “git torrent” to download the actual gigabytes of data from peers and that I would also be seeding automatically this way.
Such a tool would be nice because it would consume very little data from hugging face, while remaining as easy and native to use as it is to use git with git lfs.
> This library is a power user tool, to go beyond ~500MB/s on very high bandwidth network, where Python cannot cap out the available bandwidth.