Comment by GistNoesis | Hacker News Reader

Comment by GistNoesis | Hacker News Reader

I'm worried about the version hell of relying on HuggingFace's transformers.

In the model card : config.json [1] and generation_config.json there are strings "transformers_version": "4.28.1" and it refers to some common architecture "GPTNeoXForCausalLM" .

Which if I follow the string soft link correctly means, it refers to this file [2] with a long history of commits including some recents [3].

I don't understand how to get out of the version hell. I can install and pin a version of the transformer library, but sooner or later if I chain different foundation models I'll have to do this for each different model card that has its own version.

And then those transformer library version are probably dependent on some pytorch versions.

I don't understand how you can't reliably track a bug, or a change in behavior, or guarantee that the model that you spend millions of dollar training doesn't rust, is reproducible or become unusable due to this mess.

(And also the weights are de-serialized pickled python dictionary which is from a safety point of view equivalent to running untrusted binaries and very worrying on its own.)

[1]https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b/b... [2]https://github.com/huggingface/transformers/blob/v4.28.1/src... [3]https://github.com/huggingface/transformers/commits/v4.28.1/...

I've been diving in lately and while it's not efficient, the only way to do manage is to create a new conda/mamba environment, or a custom Docker image for all the conflicting packages.

For safety and speed, you should prefer the safetensor format: https://huggingface.co/docs/safetensors/speed

If you know what you are doing you can do your own conversions: https://github.com/huggingface/safetensors or for safety, https://huggingface.co/spaces/diffusers/convert

In terms of pickled files, we may ask them to release the models in safetensors. That would be a big improvement.

There's not much we can do about dependencies on pytorch or other python libraries. Perhaps people can make more independent implementations. The redundancy in implementations would help.

I'm surprised that some of the cutting edge AI work is not being done in pure Rust. Why even have a binding layer?

Python seems to have won over the Data / ML world.

So I’d wager they use what they and their intended audience know.

Someone implemented llama on Go recently, https://github.com/gotzmann/llama.go

Wouldn't discard a rust implementation of some LLM architecture at some point

Hi! I'm a maintainer of https://github.com/rustformers/llama-rs. We're planning to expand our model support soon.

There’s also ggml, which is quite impressive considering they get whisper and other models to run on a phone.

https://github.com/rustformers/llama-rs

The pickles drive me crazy.

Tensorflow saved models are a great way to solve the problem... Save the computation graph and weights, and drop all the crusty code dependencies. I think ONNX models are similar. I expect there should be a Jax equivalent at some point, as Jax is basically perfectly designed for this (everything is expressed in lax operations, which allows changing implementations for cpu/gpu/tpu freely... So just save the list of lax ops).

> And then those transformer library version are probably dependent on some pytorch versions.

They are not, and I dont think the model even cares about the transformers version. I run git transformers/diffusers and PyTorch 2.1 in all sorts of old repos, and if it doesnt immediately work, usually theres just small changes to APIs here and there that make scripts unhappy, and that you can manually fix.

Reply on news.ycombinator.com