If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.
7,874 karma · joined February 13, 2023
If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.
It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), it would have been more interesting.
There are some very interesting efforts in JAX/TPU land like https://github.com/erfanzar/EasyDeL
Triton does seem way more approachable as something I'd like to pick up.
In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
Was not very impressed with the chat either.
So maybe this is neat for embedded projects, but if it's Gemma only, that would be quite a sticking point for me.
Quite a thing to say as head of an AI hardware startup.
Most of the machine learning libraries we use/test in production (and that I use personally on my desktop) have hand written, extensively optimized CUDA kernels. It would be problem #1 switching to Tenstorrent hardware.
Not that I like that one bit, or that I really understand the optimization of these libraries. I do see triton code as well, but it still seems to be the suboptimal path.
I still see post of people running ollama on H100s or whatever, and that's just because its so easy to set up.
As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.
- Hybrid offloading with llama.cpp, but with slow inference.
- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.
30B-34B is more of a sweetspot for 24GB of VRAM.
If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.
If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.
Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
But Intel pretty much bet everything AI on Falcon Shores (2025). If its crumby or late, I think it will be difficult for them to catch up. They'll have inference hardware in laptops, sure, but still...
I hope y'all consider longer context models as well.
Also, are ya'll looking alternative architectures like Mamba? Being "first" with a large Mamba model would cement your architectural choices/framework support like llama did for Meta.
I run 33B on my desktop, and find it to be sufficient for many tasks.
That sounds like a chat format misconfiguration.
This could partially be Google's fault, as they used yet another novel prompting format.
Also, for sane inference speed on H100s, you'll have to wait for architecture support from the optimized frameworks. Vanilla transformers is beyond awful even with FA2.
I think so many people (including me) effectively ignored Mistral 0.1's sliding window that few realized 0.2 instruct is native 32K.
Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training" (like Solar 10.7B or Mistral 70B) and a much less intense finetuning. The former is almost like releasing a whole new base model.
It would be awesome if Mistral did that with their data, but thats very different than releasing a Gemma Instruct finetune.
Also, is the model GQA?
Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.
TBH renting a big GPU (A100 or better) and getting the run over with quick is usually best. VRAM is everything.
I am but an end user, but I noticed that disk IO for a certain app was glacial compared to a local test deployment, and I chalked it up to networking/VM overhead
Critical internet/app browsing should be taught in school, like critical reading. I feel lucky to have been a nerd in the 2000s where people picked up this skill, but honestly I have no idea how kids, older folks just getting into tech and such are acquiring sources/skills trapped inside of Discord, YouTube, Facebook or whatever.
...Which is awful.
As is often said here, Discord is where information goes to die.