My priors are like this:
1. Initial training of a neural network moves all weights around a large amount at first.
2. Later training of the network adjusts them a small amount.
3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training.
If all these things are true, then
1. Early training of an fp16 network and a bitnet with 0 added will be roughly similar in results
2. Later training will yield different / worse results, as the network gets into the 'fine tuning' part of the training.
I think the paper's stats back these priors up -- they say "this works on (3B+) large networks, but not small ones." They then imply there's something about the structure of a large network that allows a bitnet to do well. It seems more likely to me it works on large networks because they have not put the compute into 3B+ networks to get past the 'gross tuning' phase.
The networks they have compute to put in to get them 'fully' trained -- those networks don't show the results.
Also, a quick reminder that Perplexity 12 is really terrible. You would not want to use such a network. Hopefully I'm wrong and we can get something for free here! But, I'm cautious - to - skeptical.