Hacker News Reader

Comment by mikeravkine | Hacker News Reader

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ

Inference is very slow right now but it works!

View on HN

How is it possible to run this model on 2x 4090s ?

I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...

Probably this : https://huggingface.co/docs/accelerate/usage_guides/big_mode...

Without NVLink different layers are loaded onto individual cards and only one card often has to wait for the other card, slowing down generation. It's still faster than CPU offloading. You can even mix and match GPUs like a 4090 + 3060.

The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?

I think there is still lots room for improvement to reduce hardware requirements, such as 3-bit quantization or pruning weights from sparse models.

But if you're willing to spend $1500 on two used RTX 3090, it's the sweet spot in terms of the ability to run large models right now. Everything beyond that is much more expensive.

Multi-Query Attention, used here, should make 40B inference viable on systems where even 33B LLaMA with Multi-Head is basically unusable, so sometimes improvements still come from software optimization (it's no free lunch though).

The minimum size of a model with equivalent performance is unknown (like so much else about LLMs), so requirements could be reduced.

Will be interesting to see if someone comes up with an ASIC or FPGA.

Why would you think there's an ASIC/FPGA design that significantly improves over GPUs specifically targeted at running large models already? Where's the win?

The fundamental limit for hardware acceleration are number of gates you can squeeze on a die, right now. (Or, alternatively. memory bandwidth)

analog chips like what MythicAI is developing seem like the next obvious leap forward for deploying inferences broadly. ASIC/FPGA wouldn't be much different than a GPU. ASIC seems like a brittle solution

https://www.youtube.com/watch?v=GVsUOuSjvcg

This will require a cpp port first to run on e.g. apple silicon?

No, but to benefit from the ggml ecosystem, yes. Someone's taking a stab at it: https://github.com/nikisalli/falcon.cpp

This is using huggingface. It's python. There's no reason it wouldn't run on arm.

https://huggingface.co/tiiuae/falcon-40b

I'll also add that the fact something is in C++ doesn't mean it will run on arm or that it can be compiled in it.

How can it be deployed on a Huggingface Space or Colab notebook?

is there a model version where it can be deployed using huggingface's tool?

Reply on news.ycombinator.com