799 karma · joined May 14, 2015
Also, 405b has 8 KV heads of 128 size (hidden_size/num_attention_heads) times 126 layers [0] times 2 (K and V) times 2 bytes (bf16) is 504k per token. At FP8 it's 252k.
[0] https://huggingface.co/meta-llama/Meta-Llama-3.1-405B/blob/m...
Chunked prefill is for minimizing latency for decode entries in the same batch. It's not needed if you have only one request - in that case it's the fastest to just prefill in one chunk.
I'm pretty sure the sibling comment is right about different length limits - it's because of training and model talking nonsense if you let too long.
Im super impressed you got this to work! The memories of the SAMD clocks... Shudders.
Intrestingly the tiny Cortex M0+ is has about 4x the coremark per MHz of 386 (and twice 486).
See https://www.eembc.org/coremark/scores.php and https://en.wikichip.org/wiki/coremark-mhz
In Rust you could have both Iterators if it had a type parameter.
None of these are cheap though.
There are probably some reasons Rust is only non-C language to get into Linux kernel.
M3 Max is 28 TFLOPS at FP16.
Based on FLOPS alone, it would be more like a year or two.