1,103 karma · joined March 22, 2009
They may or they may not. There are schemes doing both. Either way, you cannot compare arithmetic coding given no information to your scheme that is given the exact frequencies of every symbol.
The problem is that the Shannon limit doesn't apply to your example. A fixed 1:1 ratio bit string is not I.I.D.
The article is saying this can be avoided in their system for a _microsecond_, still long enough to be potentially useful.
I think this is the most impressive part!
Temperature = 1 is where it's not pushed in either direction.
> The number of training tokens is a crucial factor for LLMs. To test the scalability of BitNet b1.58 in terms of tokens, we trained a BitNet b1.58 model with 2T tokens following the data recipe of StableLM-3B [ TBMR], which is the state-of-the-art open-source 3B model.
> [..]
> Our findings shows that BitNet b1.58 achieves a superior performance on all end tasks, indicating that 1.58-bit LLMs also have strong generalization capabilities.
The 3B model had a perplexity of 9.91, less than LLaMa 1 in fp16.
> Run LLMs locally on Cloud Workstations. Uses:
> Quantized models from [Huggingface]
> llama-cpp-python's webserver
But sure, the blog post doesn't mention it.
Are you talking about gemma.cpp? Then no, they didn't.
You need 5.4e19 artifacts before the chance is 1 in a billion.