HNHacker News
TopNewBestAskShowJobs

mmoskal

799 karma · joined May 14, 2015

https://moskal.me
submissionscomments
mmoskal··on AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
Not quite. Assume 1kW power consumption (with cooling). At $0.08/kWh (avarage US industrial rate) this is $700 per year. Adjust for more cooling etc and for say 5 years of usage but you still won't be anywhere near the $25k MSRP for H100.
mmoskal··on The Swift compiler is slow due to how types are inferred
It's funny because it's doubly exponential WRT code size. However it's also almost linear (n log*(n) due to union-find) WRT the size of the type and sane humans don't write programs with huge types.
mmoskal··on I like the RP2040
Definitely not. My understanding is they got significant market share on lunch - since it was during the pandemic chip shortage and they were the only ones with sub year lead times.
mmoskal··on Scalable MatMul-Free Language Modeling
The FPGA would be only to prove the transistor efficiency. If this works people will eventually do ASICs (possibly inside the GPUs a few generations ahead).
mmoskal··on Show HN: HackerNews but for research papers
Mobile comment view is messed up on Safari.

The top level view seems to leave a bit too much margins on mobile.

mmoskal··on Chameleon: Meta’s New Multi-Modal LLM
Assuming $30k GPU with 3yr deprecation, it's additionally $1.14/h. Much more than energy.
mmoskal··on Colorless green DNNs sleep furiously in an unexplainable fantasy
I guess the short way to say it is that "undecidable" doesn't mean "it can't ever be decided", just not always.

And of course all programs of practical significance are finite state machines (since there is only a finite number of atoms in the universe).

mmoskal··on Microsoft's Emissions Spike 29% as AI Gobbles Up Resources
One sentence from The Economist seems to explain more than TFA: "Microsoft reported a 31% increase in its indirect (Scope 3) emissions last year from building more data centres (including the carbon found in construction materials) as well as from semiconductors, servers and racks."

So no, it's not about lack of renewable electricity.

https://www.economist.com/the-world-this-week/2024/05/16/bus...

mmoskal··on Falcon 2
Projects change license for new code going forward. The old code remains available under the previous license (and sometimes new). Here, they are able to change the conditions for existing weights.
mmoskal··on Falcon 2
From model card:

Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2.

Doesn't say how long though.

mmoskal··on The Alternative Implementation Problem
The post seems to be about execution speed though. However, even there it's definitely not #1 factor as witnessed by popularity of CPython...
mmoskal··on Billions in Dirty Money Flies Under the Radar at Busiest Airports
Survivor bias? maybe they just don't catch the ones that only had two suitcases of cash.
mmoskal··on Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
TFA says you can teach it new facts, but it's very slow and makes the model hallucinate more.
mmoskal··on SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
Yes. It's speculative decoding but instead of generating just a few sequential tokens with the draft model they generate a whole tree of some sort of optimal shape with hundreds of possible sequences.

It ends up being somewhat faster than regular speculative decoding in normal setting (GPU only). If you are doing CPU offloading it's massively faster.

Edit typo

mmoskal··on Using a LLM to compress text
A problem with this is that the inference has to be 100% deterministic. For example of you change tailing order of matmul you may get slightly different results. It's easy to imagine people changing tailing in llama.cpp or other libraries. You are very unlikely to get bit exact results from different libraries.

Interestingly (though maybe not relevant here) you can also get different results from multi-user inference systems depending on what other requests are in the batch. It's possible to avoid this but I'm pretty sure most systems don't.

The "slightly different" bit of course makes it worse - it will work 99% of the time.

mmoskal··on Verified Rust for low-level systems code
Model checkers typically only explore a bounded number of states which is efficient at bug finding and often doesn't require additional annotations in the program.

Automatic (SMT-based) verifiers like Verus, Dafny, F* (and my VCC :) require you to annotate most every function and loop but give you broad guarantees about the correctness of the program.

Tools based on interactive provers (like Coq or Lean) typically require more guidance from the user but can guarantee even more complex properties.

mmoskal··on Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
Every token generates a KV cache entry based on this token and all previous KV cache entries. This happens in every layer. KV cache is 100k-1MB per token so quite a bit.

Edit: also you can forward generate 5 or 10 dots in batch without much overhead compared to a single dot since the main cost is pulling KV cache from VRAM so you have free tensor units.

mmoskal··on GitHub comments abused to push malware via Microsoft repo URLs
I guess easier to make the url be githubusercontent.com/whatever without mentioning repo name (or use uuid of repo).
mmoskal··on Meta Llama 3
Interesting, the 8B model was trained for 1.3M hours, while the 70B for 6.4M hours at 700W. Assuming $0.05/kWh (WA price) it's $46k and $224k. Even allowing for cooling, CPUs, and more expensive power wherever they are running this, still well less than $1M in power. I somehow thought it would be much more.

The nVidia bill is another matter - assuming 5 year amortization and $45k H100, it works out $1/h, so $8M or so.

mmoskal··on Lessons after a Half-billion GPT Tokens
You can ask the model sth like: is xyz correct, answer with one word, either Yes or No. The log probs of the two tokens should represent how certain it is. However, apparently RLHF tuned models are worse at this than base models.
mmoskal··on Language models are Super Mario: Absorbing abilities from homologous models
I think it's a bit like saying microcode in CPU underpins high-level programming languages - kind of, but good luck understanding what a program does based on its microcode translation. OTOH you can mess with microcode and maybe get better performance, but you won't really know before you try. It's similar with ML - much more like alchemy than science...
mmoskal··on Towards 1-bit Machine Learning Models
It seems the trick here is they first quantize it to 1- or 2-bit, and then they fine-tune the quantization bias parameters (the parameters that dequantize from 1-2 to 16 bit) via LoRA. Then they have specialized kernels to do matrix multiplication at the bit level.

Also, the 2-bit model seems much better than the 1-bit model - they use [-1, 0, 1, 2] - I wonder if '2' is needed in light of the 1.58b paper (which claims -1 is def. needed).

mmoskal··on BeagleY-AI: a 4 TOPS-capable $70 board from Beagleboard
Looks like 1/10 of TOPS and 1/3 of mem bandwidth of Jetson Orin Nano (the TI chip seems able to run 3733MT/s at 32 bit, so 15GB/s) but much cheaper.
mmoskal··on The ü/ü Conundrum
Apparently this works very well, except for a single letter, Turkish I. Turkish has two version of 'i' and Unicode folks decided to use the Latin 'i' for lowercase dotted i, and Latin 'I' for uppercase dot-less I (and have two new code points for upper-case dotted I and lower-case dot-less I).

Now, 'I'.lower() depends on your locale.

A cause for a number of security exploits and lots of pain in regular expression engines.

edit: Well, apparently 'I'.lower() doesn't depend on locale (so it's incorrect for Turkish languages); in JS you have to do 'I'.toLocaleLowerCase('tr-TR'). Regexps don't support it in neither.

mmoskal··on "Many in Silicon Valley feel the same about him (David Sacks) as you do."
This year 18/28 European NATO members will hit the 2% target. They will have to hit 3% or so even if Ukraine holds Russia off [0]. If the tanks roll to Polish and Baltic states borders it will be much more expensive.

[0] https://www.economist.com/briefing/2024/02/18/can-europe-def...

mmoskal··on Cranelift code generation comes to Rust
> JIT compilers often use techniques, such as speculative optimizations, that make it difficult to reuse the compiler outside its original context, since they encode so many assumptions about the specific language for which they were designed.

> The developers of Cranelift chose to use a more generic architecture, which means that Cranelift is usable outside of the confines of WebAssembly.

One would think this has more to do with Wasm being the source language, as it's fairly generic (compared to JS or Python), so there are no specific assumptions to encode.

Great article though. It's quite interesting to see E-matching used in compilers, took me down a memory lane (and found myself cited on Wikipedia page for e-graphs).

mmoskal··on EagleX 1.7T: Soaring past LLaMA 7B 2T in both English and Multi-lang evals
The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.
mmoskal··on The Rising Price of Power in Chips
Oh come on! No one would ever do that!... Oh wait
mmoskal··on Truffle-1 is an AI inference engine designed to run open source models at home
They say 200GB/s of mem bandwidth; Mixtral uses 13B parameters for inference; they claim 22t/s, so 22 * 13B parameters per second, so (200 * 8) bits / (22 * 13) around 5.6 bits / parameter max. With overheads, it's probably 4 bit quant.

edit: formatting

mmoskal··on Show HN: Prompts as WASM Programs
There are two parts to AICI - one is that it uses Wasm and the other is the specific set of operations on the LLM that it exposes (which is why we call it "interface").

Having a common interface (provided it is adopted) lets you re-use the same controller over multiple LLM infra stacks (and different models), or conversely the different controllers over your own brand new LLM infra stack (or model).

Using Wasm simplifies your deployment story in the cloud (sandboxing), however the AICI runtime communicates with controllers as separate processes and so if the controller is trusted it could in principle be run in native code.

← PreviousPage 4 of 10Next →