HNHacker News
TopNewBestAskShowJobs

mmoskal

799 karma · joined May 14, 2015

https://moskal.me
submissionscomments
mmoskal··on The Broken Promise of USB-C: We'll Never Get a Universal Cable
My "favorite" are these pepper grinders, milk frothers etc that have a usb c plug but only work with usb a to c cables since they do not include the two resistors that are needed to enable 900mA on usb c to c cable. And if anyone says cost reduction, SMT resistors are like 0.1 cent or less... Seems just stupid.
mmoskal··on Universe would die before monkey with keyboard writes Shakespeare, study finds
In other words the only way we know of how significant complexity arises in the universe is (ultimately) through evolution.
mmoskal··on GDDR7 Memory Supercharges AI Inference
With a typical transformer and a GPU the batch size that saturates the compute is at least hundreds. Otherwise (including typical size of 1 for local inference) you're memory bound.
mmoskal··on TSMC cuts off client after discovering chips sent to Huawei
Right some of them do. I just recall the cheapest STM32G0s not having them (though they did have OTP so you could burn it yourself).
mmoskal··on TSMC cuts off client after discovering chips sent to Huawei
All except for the cheapest microcontrollers (everything above half dollar) have serial numbers (unique identifiers) burned into one time programmable memory.
mmoskal··on Probably pay attention to tokenizers
Tokens are often sub-word, all the way down to bytes (which are implicitly understood as UTF8 but models will sometimes generate invalid UTF8...).
mmoskal··on Cuba Can't Keep the Lights On
Even the Germans didn’t make it work…
mmoskal··on Britain buys semiconductor factory for defence purposes
Don't get me started on Midwest in the US...
mmoskal··on Automatic Content Recognition Tracking in Smart TVs
I think the GP post meant that EU only wants to regulate tech if it's American.
mmoskal··on Qwen2.5: A Party of Foundation Models
You can do tensor parallelism 8 ways (8 KV heads). You can also do pipeline parallelism (there is 126 layers). Either way would work. A million tokens is possible just very slow.

Also, 405b has 8 KV heads of 128 size (hidden_size/num_attention_heads) times 126 layers [0] times 2 (K and V) times 2 bytes (bf16) is 504k per token. At FP8 it's 252k.

[0] https://huggingface.co/meta-llama/Meta-Llama-3.1-405B/blob/m...

mmoskal··on Qwen2.5: A Party of Foundation Models
Decode speed is generally memory bandwidth bound. Prefill is typically arithmetic bound. This is the reason for mixed batches (both decode and prefill) - it let's you saturate both memory and arithmetic.

Chunked prefill is for minimizing latency for decode entries in the same batch. It's not needed if you have only one request - in that case it's the fastest to just prefill in one chunk.

I'm pretty sure the sibling comment is right about different length limits - it's because of training and model talking nonsense if you let too long.

mmoskal··on Exploring the scalable matrix extension of the Apple M4 processor
IIUC this the cpu in an iPad. The pro/max versions would be more appropriate to compare against the Zen when they are released.
mmoskal··on My business card runs Linux and Ultrix (2022)
Also M33 vs M0 and higher clock - should be 2-3x faster. And maybe you can do something with the second core.

Im super impressed you got this to work! The memories of the SAMD clocks... Shudders.

mmoskal··on My business card runs Linux and Ultrix (2022)
It's software emulated though. The actual speed is equivalent to around 1MHz.

Intrestingly the tiny Cortex M0+ is has about 4x the coremark per MHz of 386 (and twice 486).

See https://www.eembc.org/coremark/scores.php and https://en.wikichip.org/wiki/coremark-mhz

mmoskal··on Synthetic diamonds are now purer, more beautiful, and cheaper than mined
Literally The Diamond Age.
mmoskal··on Windows NT vs. Unix: A design comparison
Interestingly in a Javascript or similar runtime most of text that hits the caches where the size actually matters is still ASCII even in far east because of identifiers. Utf8 for the win!
mmoskal··on Operating system threads are always going to be (more) expensive
macOS has futexes it just doesn't talk about them... https://crates.io/crates/ulock-sys
mmoskal··on Hardware Acceleration of LLMs: A comprehensive survey and comparison
It's a different kind of memory chip that also does some computation. See https://en.m.wikipedia.org/wiki/In-memory_processing
mmoskal··on Achieving high Python performance with code generation (2022)
Typical speed difference between Python and native is around two orders of magnitude (of course for code that actually does computation in Python, not just call numpy etc).
mmoskal··on Fear Not the Association of Types
In Java you can't have a class implement both Iterator<Foo> and Iterator<Bar> (type erasure; or maybe my Java is out of date...). Anyway if you could, the for loop would not know which to use.

In Rust you could have both Iterators if it had a type parameter.

mmoskal··on Porting my JavaScript game engine to C for no reason
The way I heard it described is that you call the library while the framework calls you.
mmoskal··on USB Sniffer Lite for RP2040
I successfully used Beagle USB 12 and 480 https://www.totalphase.com/products/beagle-usb480/ . Saleae https://www.saleae.com also has some USB functions but not as extensive. It's much more versatile and look cooler though :)

None of these are cheap though.

mmoskal··on Investing in Rust
Which alternatives exactly? Mainstream languages like Go and Swift use a GC and don't allow the same level of control over memory layout.

There are probably some reasons Rust is only non-C language to get into Linux kernel.

mmoskal··on Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c
Right... and there are probably some communication overheads over NVLink that would not be present on single laptop. So a few months maybe :)
mmoskal··on Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c
H100 SXM is 2000 TFLOPS at FP16. Multiply by 8.

M3 Max is 28 TFLOPS at FP16.

Based on FLOPS alone, it would be more like a year or two.

mmoskal··on How Good Is ChatGPT at Coding, Really?
Paper link https://arxiv.org/abs/2308.04838
mmoskal··on Speeding up the JavaScript ecosystem – Isolated Declarations
You would be probably better off just running esbuild or similar - it will happily strip types from any version of TS.
mmoskal··on Speeding up the JavaScript ecosystem – Isolated Declarations
I think most bundlers can handle TS nowadays so you don't actually need tsc.
mmoskal··on The Diamond Sutra, the oldest dated printed book (2016)
Everything that is experienced except for awareness itself.
mmoskal··on Delving into ChatGPT usage in academic writing through excess vocabulary
LLMs used in writing of 1/3 of scientific papers. On the one hand scary, on the other it seems people find them useful. Maybe this AI thing is not like crypto after all...
← PreviousPage 3 of 10Next →