420 karma · joined January 10, 2025
For text generation, which is the most important metric, the tokens per second will scale almost linearly with memory bandwidth (936 GB/s, 1008 GB/s and 1792 GB/s respectively), but we might see more interesting results when comparing prompt processing, speculative decoding with various models, vLLM vs llama.cpp vs TGI, prompt length, context length, text type/programming language (actually makes a difference with speculative decoding), cache quantization and sampling methods. Results should also be checked for correctness (perplexity or some benchmark like HumanEval etc.) to make sure that results are not garbage.
If anyone from Phoronix is reading this, this post might be a good point to get you started: https://old.reddit.com/r/LocalLLaMA/comments/1h5uq43/llamacp...
At time of writing, Qwen2.5-Coder-32B-Instruct-GGUF with one of the smaller variants for speculative decoding is probably the best local model for most programming tasks, but keep an eye out for any new models. They will probably show up in Bartowksi's "Recommended large models" list, which is also a good place to download quantized models: https://huggingface.co/bartowski
See e.g. https://github.com/99991/dynamic/blob/a423a04061ee44bad0720f... for an example (incidentally also a C JSON parser, but with automatic garbage collection).
For C, you can use the switch -Wswitch-enum with GCC to get warnings and add -Werror to turn the warning into an error.
1. LLMs "think" in terms of tokens, which usually are around 4 characters each. While humans only have to memorize 10x10 multiplication tables to perform multiplication of large numbers, LLMs have to memorize a 10000x10000 table, which is much more difficult.
2. LLMs can't "think in their head", so you have to make them spell out each step of the multiplication, just like (most) humans can't multiply huge numbers without intermediate steps.
A simple way to demonstrate this is to ask an LLM for the birth year of a celebrity and then whether that number is even or odd. The answer will be correct almost every time. But if you ask whether the birth year of a celebrity is even or odd and forbid spelling out the year, the accuracy will be barely above 50 %.
https://picryl.com/media/girl-portrait-looking-people-c182c0
(Screenshot of website in case OP deletes it https://i.imgur.com/Fv7w9p4.png )
Here are a few cats (not cherry picked, no postprocessing): https://files.catbox.moe/5zpp78.png
Some images are not quite perfect, but if you generate enough, there will probably be a good one among them.
I've also heard of a fun database where the user "Geoffrey" was truncated to "G". I wonder why? (Hint: End of file)
https://huggingface.co/spaces/Qwen/Qwen2-VL
You can also run a smaller model locally if you have enough VRAM, for example Qwen2.5-VL-7B-Instruct:
https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file#usin...
Also works reasonably well with hand-written equations.
For searching through similar equations, you can probably embed each as a high-dimensional vector and then search for the closed vector. Here is a ranking of text embedding networks:
https://huggingface.co/spaces/mteb/leaderboard
Or if you want something more deterministic, parse the LaTeX equation to create an abstract syntax tree for which there are plenty of similarity measures.
Unfortunately, this is only deterministic on the same hardware, but there is no reason why one couldn't write reasonably efficient LLM kernels. It just has not been a priority.
Nevertheless, I still agree with the main point that it is difficult to get LLMs to produce the same output reliably. A small change in the context might trigger all kinds of changes in the generated code.