799 karma · joined May 14, 2015
The top level view seems to leave a bit too much margins on mobile.
And of course all programs of practical significance are finite state machines (since there is only a finite number of atoms in the universe).
So no, it's not about lack of renewable electricity.
https://www.economist.com/the-world-this-week/2024/05/16/bus...
Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2.
Doesn't say how long though.
It ends up being somewhat faster than regular speculative decoding in normal setting (GPU only). If you are doing CPU offloading it's massively faster.
Edit typo
Interestingly (though maybe not relevant here) you can also get different results from multi-user inference systems depending on what other requests are in the batch. It's possible to avoid this but I'm pretty sure most systems don't.
The "slightly different" bit of course makes it worse - it will work 99% of the time.
Automatic (SMT-based) verifiers like Verus, Dafny, F* (and my VCC :) require you to annotate most every function and loop but give you broad guarantees about the correctness of the program.
Tools based on interactive provers (like Coq or Lean) typically require more guidance from the user but can guarantee even more complex properties.
Edit: also you can forward generate 5 or 10 dots in batch without much overhead compared to a single dot since the main cost is pulling KV cache from VRAM so you have free tensor units.
The nVidia bill is another matter - assuming 5 year amortization and $45k H100, it works out $1/h, so $8M or so.
Also, the 2-bit model seems much better than the 1-bit model - they use [-1, 0, 1, 2] - I wonder if '2' is needed in light of the 1.58b paper (which claims -1 is def. needed).
Now, 'I'.lower() depends on your locale.
A cause for a number of security exploits and lots of pain in regular expression engines.
edit: Well, apparently 'I'.lower() doesn't depend on locale (so it's incorrect for Turkish languages); in JS you have to do 'I'.toLocaleLowerCase('tr-TR'). Regexps don't support it in neither.
[0] https://www.economist.com/briefing/2024/02/18/can-europe-def...
> The developers of Cranelift chose to use a more generic architecture, which means that Cranelift is usable outside of the confines of WebAssembly.
One would think this has more to do with Wasm being the source language, as it's fairly generic (compared to JS or Python), so there are no specific assumptions to encode.
Great article though. It's quite interesting to see E-matching used in compilers, took me down a memory lane (and found myself cited on Wikipedia page for e-graphs).
edit: formatting
Having a common interface (provided it is adopted) lets you re-use the same controller over multiple LLM infra stacks (and different models), or conversely the different controllers over your own brand new LLM infra stack (or model).
Using Wasm simplifies your deployment story in the cloud (sandboxing), however the AICI runtime communicates with controllers as separate processes and so if the controller is trusted it could in principle be run in native code.