Hacker News Reader

The 176B parameter BLOOM model should be available soon: https://bigscience.notion.site/BLOOM-BigScience-176B-Model-a...

yes, looking forward to it especially because it is going to be multilingual by design

Dumb question: does 175B parameters mean the number of bytes (or floats?) in the model? Does that also mean you need the whole model in memory to do inference (in practice)?

If so, not many machines have that much RAM. Makes it hard to "play" with.

float16s or bfloat16s so 2x of that for storage

You can infer using DeepSpeed.

>However, they did not publish any details on their training data.

Yes they did. It's in the README.

All I can see is “ 1.7 TB of online texts, books, and countless other sources in both English and Russian.”

if there are more details, can you please share a link?

I am worried that “other sources” may contain Yandex.news which is a cesspool of anti-West and anti-Ukraine propaganda

The pile dataset is used for the English language

Direct link: https://github.com/yandex/YaLM-100B/blob/main/README.md#trai...