Hacker News Reader

Comment by dekhn | Hacker News Reader

If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.

View on HN

You can actually download the weights without an account:

    git clone https://huggingface.co/bigscience/bloom

You will need Git LFS and ~330GB of free space though.

It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then:

    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM
    
    tokenizer = AutoTokenizer.from_pretrained('./bloom')
    inputs = tokenizer("Hacker News is", return_tensors="pt")
    model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16)
    
    output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True)
    print(tokenizer.decode(output[0].tolist()))

After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk):

    print(model.hf_device_map)

Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs:

https://twitter.com/moyix/status/1546961979566325760

Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working.

Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec

Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.

I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB:

    $ du -csh pytorch_model*.bin | tail -n 1
    329G    total
    $ du -sh .git 
    329G    .git

Not sure if there's a way to get git-lfs not to do that...

I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).

If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?

Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU.

I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issueco...

I wasn't going to ask about this until I had it running on my machine but... its very interesting that you can do out of core models with weights in various memory hierarchies. It feels almost like swap/paging. This is exciting!

Simple swap via memory-mapped files would have worked probably. The issue is the lack of prefetch.

Would NVLink and more RAM improve performance?

Almost there. I've got everything installed and want to run your inference script. Model loading fails, TypeError: Got unsupported ScalarType BFloat16

  File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/utils/offload.py", line 25, in offload_weight
    array = weight.numpy()

I verified my numpy knows nothing of bfloat16:

>>> import torch >>> t=torch.arange(0,6).resize(2,3).to(dtype=torch.bfloat16) /home/dek/miniconda3/lib/python3.9/site-packages/torch/_tensor.py:586: UserWarning: non-inplace resize is deprecated warnings.warn("non-inplace resize is deprecated") >>> t tensor([[0., 1., 2.], [3., 4., 5.]], dtype=torch.bfloat16) >>> t.numpy() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Got unsupported ScalarType BFloat16

Seems par for the course that I'd be able to get the world's most advanced NLP model downloaded over gigabit to my home supercomputer only to be stymied by a data type.

Comment three: after hacking my way around that, the script ran for another few hours doing disk IE (writing the offload) such that my directory is now 1.3T (330GB .git/lfs/objects, 330GB sharded weights, and 607G of "offload".

Then it failed again:

    File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/big_modeling.py", line 188, in dispatch_model
      main_device = [d for d in device_map.values() if d not in ["cpu", "disk"] [0]
  IndexError: list index out of range

Now I'm curious just how long it will take to repro this error (IE, running again, with the offload files aalready written). It's also puzzling since I set the device_map to 'auto'.

Again, all par for the course and stuff I expected (having worked in HPC/ML/science for 3 decades, you get used to research codes).

Huh. That one looks like it wasn't able to place any part of the model on the GPU. I know there is currently an issue where if the first layer of the model is too big to fit on the GPU it will put all layers on the CPU/disk (rather than trying to see if a later layer would fit). But the 3080Ti has 12GB so I'd be surprised if that's happening here?

I see the same error as well. Looking at the underlying accelerate code, it looks like if GPU is not an option then CPU and disc are completely rejected and the above error is thrown.

Just to double-check, are you using the git versions of transformers and accelerate? I believe the BFloat16 support is pretty recent (22 days ago) and may not be in conda/pip:

https://github.com/huggingface/accelerate/commit/f13c59f91e2...

What does 'offload_folder' parameter do? Why is it even needed? Does the inference produce intermediate results so large they have to be stored on disk? o-O

You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio 13:23

Reply on news.ycombinator.com