Run Llama 2 locally using xet mount, from 0 to generating text in < 10M

We were excited to see Meta release Llama 2 yesterday, but found it a little annoying to download the 6 models (331GB, took us ~4 hours from an EC2 instance with 10GB network) to start playing with them. So we brought them into XetHub and now have them available for use here: https://xethub.com/XetHub/Llama2.

By using xet mount you can get started in seconds, and within a few minutes have the model generating text, without needing to download everything or make an inference API call.

  # From a g4dn.8xlarge instance in us-west-2:
  
  Mount complete in 8.629213s
  
  # install model requirements, and then ...
  
  (venv-test) ubuntu@ip-10-0-30-1:~/Llama2/code$ torchrun --nproc_per_node 1 example_chat_completion.py \
      --ckpt_dir ../models/llama-2-7b-chat/ \
      --tokenizer_path ../models/tokenizer.model \
      --max_seq_len 512 --max_batch_size 4
  > initializing model parallel with size 1
  > initializing ddp with size 1
  > initializing pipeline with size 1
  
  Loaded in 306.17 seconds
  
  User: what is the recipe of mayonnaise?
  
  > Assistant:  Thank you for asking! Mayonnaise is a popular condiment made from a mixture of egg yolks, oil, vinegar or lemon juice, and seasonings. Here is a basic recipe for homemade mayonnaise:
  
  ...

Check out the repo README for detailed instructions, and remember to register with Meta to accept the license and acceptable use policy for these models.

Xet mount performance is pretty decent now, but it’s still WIP and we are making significant optimizations. Our storage architecture is massively scalable. See this blog post for more details: https://about.xethub.com/blog/mount-part-1.

Happy Llama 2-ing! We’ll be adding the -GGML variants for those using llama.cpp next, hopefully later today.