HNHacker News
TopNewBestAskShowJobs

threeducks

420 karma · joined January 10, 2025

submissionscomments
threeducks··on Llama.cpp AI Performance with the GeForce RTX 5090 Review
Performance for text generation is memory-limited, so lack of native fp8 support does not matter. You have more than enough compute left over to do the math in whichever floating point format you fancy.
threeducks··on Llama.cpp AI Performance with the GeForce RTX 5090 Review
I agree. This benchmark should have compared the largest ~4 bit quantized model that fits into VRAM, which would be somewhere around 32B for RTX 3090/4090/5090.

For text generation, which is the most important metric, the tokens per second will scale almost linearly with memory bandwidth (936 GB/s, 1008 GB/s and 1792 GB/s respectively), but we might see more interesting results when comparing prompt processing, speculative decoding with various models, vLLM vs llama.cpp vs TGI, prompt length, context length, text type/programming language (actually makes a difference with speculative decoding), cache quantization and sampling methods. Results should also be checked for correctness (perplexity or some benchmark like HumanEval etc.) to make sure that results are not garbage.

If anyone from Phoronix is reading this, this post might be a good point to get you started: https://old.reddit.com/r/LocalLLaMA/comments/1h5uq43/llamacp...

At time of writing, Qwen2.5-Coder-32B-Instruct-GGUF with one of the smaller variants for speculative decoding is probably the best local model for most programming tasks, but keep an eye out for any new models. They will probably show up in Bartowksi's "Recommended large models" list, which is also a good place to download quantized models: https://huggingface.co/bartowski

threeducks··on Skynet won and destroyed humanity
The dangers of "model collapse" are wildly overstated. Sure, if you feed the unfiltered output of an LLM into a new LLM, inbreeding will eventually collapse the model, but if you filter the data with some grounding in reality, the results will get better and better. The best example is probably AlphaGo Zero, which was grounded with the rules of the game and did not even receive any supervised data. For programming, grounding can happen by simply executing the code. But even if you do not have grounding, you can still just throw a lot of test time compute at the output to filter out garbage, which is probably good enough.
threeducks··on Abusing C to implement JSON parsing with struct methods
You could tag dynamically allocated objects with some magic bytes. To then check whether a string is dynamically allocated, you can simply compare the first few bytes against the magic bytes. This is a probabilistic method, but if you use enough magic bytes, it becomes basically impossible to go wrong.

See e.g. https://github.com/99991/dynamic/blob/a423a04061ee44bad0720f... for an example (incidentally also a C JSON parser, but with automatic garbage collection).

threeducks··on A possible future of Python in the browser
> it was the first language that ever gave me compile error because there were cases that my switch statement didn't account for.

For C, you can use the switch -Wswitch-enum with GCC to get warnings and add -Werror to turn the warning into an error.

threeducks··on Why don't LLMs ask for calculators?
Yes, but we could also give the LLM access to a Python interpreter and solve a much larger class of problems with correctness guarantees and around a billion times less compute.
threeducks··on Why don't LLMs ask for calculators?
Two reasons:

1. LLMs "think" in terms of tokens, which usually are around 4 characters each. While humans only have to memorize 10x10 multiplication tables to perform multiplication of large numbers, LLMs have to memorize a 10000x10000 table, which is much more difficult.

2. LLMs can't "think in their head", so you have to make them spell out each step of the multiplication, just like (most) humans can't multiply huge numbers without intermediate steps.

A simple way to demonstrate this is to ask an LLM for the birth year of a celebrity and then whether that number is even or odd. The answer will be correct almost every time. But if you ask whether the birth year of a celebrity is even or odd and forbid spelling out the year, the accuracy will be barely above 50 %.

threeducks··on Are SSDs more reliable than hard drives? (2021)
I've had the opposite experience. An SSD died without any warning resulting in complete data loss, while a dying HDD started making funny noises, which prompted the thought "Hmm, might be time for a backup again".
threeducks··on Writing a Gimp 3.0 Plugin
I often hear that Krita has a different focus, but does it lack anything that GIMP has? (disregarding obscure features like bell pepper brush)
threeducks··on Show HN: I built an AI coloring page generator
Ironically, their terms of service forbid the generation of copyrighted content.
threeducks··on Show HN: I built an AI coloring page generator
Those testimonials are AI-generated and fake. For example, "Emma Davis" is this stock photo:

https://picryl.com/media/girl-portrait-looking-people-c182c0

(Screenshot of website in case OP deletes it https://i.imgur.com/Fv7w9p4.png )

threeducks··on Show HN: I built an AI coloring page generator
The free Flux.1 Dev model with this LoRA can do it. https://civitai.com/models/794953/coloring-book-flux

Here are a few cats (not cherry picked, no postprocessing): https://files.catbox.moe/5zpp78.png

Some images are not quite perfect, but if you generate enough, there will probably be a good one among them.

threeducks··on Nvidia's RTX 5090 power connectors are melting
It boggles the mind how a $10 space heater has better overheating protection than this $4000 space heater.
threeducks··on Hello, I'm Mr. Null. My Name Makes Me Invisible to Computers (2015)
Sorry, I don't have first-hand proof. But it seems that this issue is not rare: https://old.reddit.com/r/programminghorror/comments/4g70lj/s...
threeducks··on Hello, I'm Mr. Null. My Name Makes Me Invisible to Computers (2015)
A common source of "NULL" strings are CSVs. I personally had to deal with a database that only understood NULLs correctly when using QUOTE_NOTNULL, which was added to Python just last year: https://github.com/python/cpython/issues/113732

I've also heard of a fun database where the user "Geoffrey" was truncated to "G". I wonder why? (Hint: End of file)

threeducks··on Large Language Models for Mathematicians (2023)
Visual language models are quite good at converting images to LaTeX or any other kind of representation these days. Here is a demo where you can upload a screenshot of a page with an equation and ask the neural network to e.g. "Transcribe this equation as LaTeX.":

https://huggingface.co/spaces/Qwen/Qwen2-VL

You can also run a smaller model locally if you have enough VRAM, for example Qwen2.5-VL-7B-Instruct:

https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file#usin...

Also works reasonably well with hand-written equations.

For searching through similar equations, you can probably embed each as a high-dimensional vector and then search for the closed vector. Here is a ranking of text embedding networks:

https://huggingface.co/spaces/mteb/leaderboard

Or if you want something more deterministic, parse the LaTeX equation to create an abstract syntax tree for which there are plenty of similarity measures.

threeducks··on Tabby: Self-hosted AI coding assistant
You can set PyTorch to deterministic mode with a small performance penalty: https://pytorch.org/docs/stable/notes/randomness.html#avoidi...

Unfortunately, this is only deterministic on the same hardware, but there is no reason why one couldn't write reasonably efficient LLM kernels. It just has not been a priority.

Nevertheless, I still agree with the main point that it is difficult to get LLMs to produce the same output reliably. A small change in the context might trigger all kinds of changes in the generated code.

threeducks··on A three month review of kagi search and the orion web browser (2024)
Less than half of HN users are from the US and wages are lower in most countries, sometimes by a lot. Less than 10% in Turkey or Ukraine for example: https://en.wikipedia.org/wiki/List_of_countries_by_average_w...
← PreviousPage 5 of 5