HNHacker News
TopNewBestAskShowJobs

mmoskal

799 karma · joined May 14, 2015

https://moskal.me
submissionscomments
mmoskal··on Bitter Lesson is about AI agents
The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effort, as it becomes compartively cheaper.

Also the current crop of models are inherently limited. Even for something as simple as following a JSON schema, models alone are not good enough [0]

Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x.

[0] https://openai.com/index/introducing-structured-outputs-in-t... - see plot

mmoskal··on Oxidizing Ubuntu: adopting Rust utilities by default
The vec pattern has some advantages though, in particular you can often get away with 32 bit indices (instead of 64 bit pointers), making it a little more cache-friendly. I did it for regex ASTs that are supposed to be hash-consed, so never need to die until the whole matcher dies [0].

A more contrived example is Earley items in parser that point inside a grammar rule (all rules are in flat vec) and back into previous parser state - so I have 2 u32 offsets. If I had pointers, I would be tempted to have a pointer to grammar rule, index inside of it, and pointer to previous state [1], so probably 3x the space.

In both cases, pointers would be easier but slower. Annoying that Rust doesn't really let you make the choice though...

[0] https://github.com/microsoft/derivre/blob/main/src/hashcons.... [1] https://github.com/guidance-ai/llguidance/blob/main/parser/s...

mmoskal··on Orpheus-3B – Emotive TTS by Canopy Labs
I wonder how can it be Apache if it's based on Llama?
mmoskal··on TensorRT-LLM runtime now open-source
Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).
mmoskal··on Effective Rust (2024)
This comes from ML (as in SML or OCaml), where 'a reads "alpha" and is a type parameter.
mmoskal··on LeetCode but You Can Force People to Code in Light Mode
I think universally light when using a projector! But maybe nowadays it's remote or big LCDs...
mmoskal··on Google Titans Model Explained: The Future of Memory-Driven AI Architectures
My favorites are Nvidia Triton (server) and Openai Triton (compiler)
mmoskal··on DeepSeek Open Infra: Open-Sourcing 5 AI Repos in 5 Days
Their tech report says one inference deployment is around 400 GPUs...
mmoskal··on 21st Century C++
> ACM, the Association for Computing Machinery, is the world's largest educational and scientific society, uniting computing educators, researchers and professionals to inspire dialogue, share resources and address the field's challenges.

Most of programming language conferences are organized by ACM.

mmoskal··on Why DeepSeek had to be open source
You're most likely running a destilled version. The full model is ~700GB.
mmoskal··on Searching for DeepSeek's glitch tokens
You essentially have to run a byte regular expression that enforces valid UTF8. When you take into account exclusion for surrogate pairs and overlongs, you end up with about 14 states in the corresponding automaton.

This is one thing among many done by our llguidance [0] library.

[0] https://github.com/microsoft/llguidance

edit: if anyone's interested:

(([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC] [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF] [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF] [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])

mmoskal··on Searching for DeepSeek's glitch tokens
llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens with UTF8 fragments in llama3).

There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.

mmoskal··on Searching for DeepSeek's glitch tokens
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8).

Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.

mmoskal··on DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
Luckily in the US the govt can do no such things due to the 1st amendment, so it only takes a relevant billionaire to get a model with different political views.
mmoskal··on DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
CCP requires models to follow "socialist values".

https://www.cnbc.com/amp/2024/07/18/chinese-regulators-begin...

mmoskal··on Using ChatGPT is not bad for the environment
737-800 burns about 3t of fuel per hour. NYC-SFO is about 6h, so 18t of fuel. Jet fuel energy density is 43MJ/kg, so 774000 MJ per flight, which is 215 MWh. Assuming the 60 GWh figure is true (seems widely cited on the internets), it comes down to 279 one-way flights.
mmoskal··on Using ChatGPT is not bad for the environment
I assume this comes from the 60GWh figure, which does translate to about 200 flights (assuming energy density of gasoline; in actual CO2 emissions it was probably less since likely cleaner energy was used than for running planes).
mmoskal··on Nvidia might do for desktop AI what it did for desktop gaming
I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take.

For comparison H100 can read its memory 40 times per second, so if you use it all you can get around 40t/s.

Of course in either case you don't have to fill it up, but instead use smaller model or more GPUs.

mmoskal··on Structured Outputs with Ollama
Models are trained on Markdown, JSON and various programming languages, so either one of these should work.

However, in this case, you're best of giving the model sentences one by one to avoid it being confused. If you structure the prompt like "Classify the following sentence, here are the rules ...." + sentence, then you should be hitting prefix cache and get even better performance than when doing a single query. Of course, this only works if you have the prefix cache and are not paying per input token (though most providers now let you indicate you want to use prefix cache and pay less).

mmoskal··on Structured Outputs with Ollama
It depends how fine-tuned the model is to JSON output.

Also, you need to tell the model the schema. If you don't you will get more weird tokenization issues.

For example, if the schema expects a JSON key "foobarbaz" and the canonical BPE tokenization is ["foobar", "baz"], the token mask generated by all current constrained output libraries will let the model choose from "f", "foo", "foobar" (assuming these are all valid tokens). The model might then choose "foo", and then the constraint will force eg. "bar" and "baz" as next tokens. Now the model will see ["foo", "bar", "baz"] instead of ["foobar", "baz"] and will get confused [0]

If the model knows from the prompt "foobarbaz" is one of the schema keys, it will generally prefer "foobar" over "foo".

[0] In modern models these tokens are related, because of regularization but they are not the same.

mmoskal··on Structured Outputs with Ollama
If you want avoid startup cost, llguidance [0] has no compilation phase and by far the fullest JSON support [1] of any library. I did a PoC llama.cpp integration [2] though our focus is mostly server-side [3].

[0] https://github.com/guidance-ai/llguidance [1] https://github.com/guidance-ai/llguidance/blob/main/parser/s... [2] https://github.com/ggerganov/llama.cpp/pull/10224 [3] https://github.com/guidance-ai/llgtrt

mmoskal··on Structured Outputs with Ollama
It seems ollama only supports JSON Schema.

Interestingly, JSON Schema has much less of this problem than say CSV - when the model is forced to produce `{"first_key":` it will generally understand it's supposed to continue in JSON. It still helps to tell it the schema though, especially due to weird tokenization issues you can get otherwise.

mmoskal··on How We Optimize LLM Inference for AI Coding Assistant
> The academic literature has recently caught on to this approach as well and we want to highlight the excellent papers on Sarathi (link 1, link 2) and DeepSpeed-FastGen (link). The academic literature calls the technique “chunked prefill”.

The Sarathi paper is more than a year old and implemented in both vllm and trtllm. There doesn't seem to be much else of substance in the article. Edit: to be fair it seems the article was written some time ago and only "updated" two weeks ago. So maybe it was more novel at the time.

mmoskal··on Understanding SIMD: Infinite complexity of trivial problems
I imagine there is and order of magnitude of difference between how much you can translate in software, with large memory and significant time budget to work with, compared to microcode.
mmoskal··on Charset="WTF-8"
Abstract

WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.

mmoskal··on El Capitan: New supercomputer is the fastest
2Tflops was #500 in 2006. Still a supercomputer. Crazy.

https://www.top500.org/statistics/perfdevel/

mmoskal··on Show HN: Jaws – a JavaScript to WASM ahead-of-time compiler
Back in the day I did an almost-Typescript (though much closer than assembly script) to embedded ARM compiler. Some of the techniques may be useful.

https://www.microsoft.com/en-us/research/uploads/prod/2019/0...

mmoskal··on Algol-68 seemed like a good idea
B came from BCPL which came from CPL which came from ALGOL 60. The main thing about ALGOL was structured programming as opposed to goto statements. Compared to that begin/end vs braces is a very minor issue.
mmoskal··on DeepMind debuts watermarks for AI-generated text
The most typical reason argmax (temp 0) is non-deterministic is that your request is running batched with other people requests. The number and size of these affects the matrix sizes and thus tiling decisions. Then you get different floating point order and thus different results.

Nvidia gives some guarantees about deterministic results of their kernels but that only applies when you have exact same input data and this is not the case when in-flight batching.

mmoskal··on DeepMind debuts watermarks for AI-generated text
They actually run 2^30-way tournament (they derive an equivalent form that doesn't requires 2B operations). You do not need to run the LLM, it only depends on the tokenizer.
← PreviousPage 2 of 10Next →