You create a hugging face token, set an env variable (HF_TOKEN) and download it with the hugging face CLI or just automatically as needed with trandormers, vllm, etc…
Previously with non-gated models, you could download the XET files with any program, no special software needed.
Now I have wasted some time until finding that I can use curl with the options:
until curl --user "name" --oauth2-bearer "access_token" --location --continue-at - "HF_XET_link"; do sleep xx; done
Previously I was using wget, which was very reliable, which is important for a download that may take a day or even more.
Now I must use curl due to the authentication method and curl sucks badly at retrying and resuming interrupted transfers. Therefore I must write in the shell script a loop that tests the exit code and re-executes curl with the --continue-at option until the download is successful. Ugly in comparison with wget.
The command written above is a simplified loop. A better loop would need to test specific exit error codes of curl, but discovering which are those would need time for searching the curl documentation or sources.