Hacker News Reader

Comment by PoignardAzur | Hacker News Reader

You can try the model on HuggingFace: https://huggingface.co/bigscience/bloom

Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model).

Bloom looks pretty exciting. It's reportedly as performant as GPT-3 (I haven't tested it enough to confirm, but what little testing I did gave okay results), but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil.

I'm expecting the coming 12 months to be pretty interesting for text generation.

View on HN

> but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil

It seems that the developers' position is that it's not fully open-source, and that some potentially evil or unethical uses are actually restricted:

https://huggingface.co/spaces/bigscience/license

Right, it's open source with a license that forbids using it for evil.

Practically speaking, though, you can just download it on a private server and do evil and afaict nobody can stop you or know you're using BLOOM.

> Right, it's open source with a license that forbids using it for evil.

That's not considered open source, though.

https://opensource.org/osd-annotated

Also compare

https://web.archive.org/web/20130203112329/http://dev.hasenj...

(although IBM wasn't asking specifically about the open source nature of the license).

Alright, fine. It's open-source-with-an-asterisk, which is completely different from open-source in every way, and happens to involve having access to the source of the thing.

The asterisk says "I'm not allowed to say it's open source because some website I don't care about says it's not the right definition".

> Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model).

This model is larger than anything HF has done, so resource usage likely has to be constrained a bit (GPT-3 was restricted at first for similar reasons, in addition to the potential abuse issues)

If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.

You can actually download the weights without an account:

    git clone https://huggingface.co/bigscience/bloom

You will need Git LFS and ~330GB of free space though.

It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then:

    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM
    
    tokenizer = AutoTokenizer.from_pretrained('./bloom')
    inputs = tokenizer("Hacker News is", return_tensors="pt")
    model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16)
    
    output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True)
    print(tokenizer.decode(output[0].tolist()))

After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk):

    print(model.hf_device_map)

Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs:

https://twitter.com/moyix/status/1546961979566325760

Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working.

Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec

Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.

I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB:

    $ du -csh pytorch_model*.bin | tail -n 1
    329G    total
    $ du -sh .git 
    329G    .git

Not sure if there's a way to get git-lfs not to do that...

I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).

If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?

Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU.

I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issueco...

I wasn't going to ask about this until I had it running on my machine but... its very interesting that you can do out of core models with weights in various memory hierarchies. It feels almost like swap/paging. This is exciting!

Simple swap via memory-mapped files would have worked probably. The issue is the lack of prefetch.

Would NVLink and more RAM improve performance?

Almost there. I've got everything installed and want to run your inference script. Model loading fails, TypeError: Got unsupported ScalarType BFloat16

  File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/utils/offload.py", line 25, in offload_weight
    array = weight.numpy()

I verified my numpy knows nothing of bfloat16:

>>> import torch >>> t=torch.arange(0,6).resize(2,3).to(dtype=torch.bfloat16) /home/dek/miniconda3/lib/python3.9/site-packages/torch/_tensor.py:586: UserWarning: non-inplace resize is deprecated warnings.warn("non-inplace resize is deprecated") >>> t tensor([[0., 1., 2.], [3., 4., 5.]], dtype=torch.bfloat16) >>> t.numpy() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Got unsupported ScalarType BFloat16

Seems par for the course that I'd be able to get the world's most advanced NLP model downloaded over gigabit to my home supercomputer only to be stymied by a data type.

Comment three: after hacking my way around that, the script ran for another few hours doing disk IE (writing the offload) such that my directory is now 1.3T (330GB .git/lfs/objects, 330GB sharded weights, and 607G of "offload".

Then it failed again:

    File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/big_modeling.py", line 188, in dispatch_model
      main_device = [d for d in device_map.values() if d not in ["cpu", "disk"] [0]
  IndexError: list index out of range

Now I'm curious just how long it will take to repro this error (IE, running again, with the offload files aalready written). It's also puzzling since I set the device_map to 'auto'.

Again, all par for the course and stuff I expected (having worked in HPC/ML/science for 3 decades, you get used to research codes).

Huh. That one looks like it wasn't able to place any part of the model on the GPU. I know there is currently an issue where if the first layer of the model is too big to fit on the GPU it will put all layers on the CPU/disk (rather than trying to see if a later layer would fit). But the 3080Ti has 12GB so I'd be surprised if that's happening here?

I see the same error as well. Looking at the underlying accelerate code, it looks like if GPU is not an option then CPU and disc are completely rejected and the above error is thrown.

Just to double-check, are you using the git versions of transformers and accelerate? I believe the BFloat16 support is pretty recent (22 days ago) and may not be in conda/pip:

https://github.com/huggingface/accelerate/commit/f13c59f91e2...

What does 'offload_folder' parameter do? Why is it even needed? Does the inference produce intermediate results so large they have to be stored on disk? o-O

You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio 13:23

Reply on news.ycombinator.com