They make a new booster about every six weeks.
333 karma · joined October 14, 2020
Before that last flight it seemed like engine reliability wouldn't get solved. But now they have a clear path to orbit.
I doubt the heat shields will hold up on the first several re-entry attempts but there's been recent sightings of a new heat shield that has tiles about half the size. Maybe smaller tiles will have less lever action and will have 4x the fasteners.
Official post on Mixtral 8x7B: https://mistral.ai/news/mixtral-of-experts/
Official PR into vLLM shows the inference code: https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6...
New HuggingFace explainer on MoE very nice: https://huggingface.co/blog/moe
In naive decoding, performance of a bit above 70B (Llama 2), at inference speed of ~12.9B dense model (out of total 46.7B params).
Notes: - Glad they refer to it as "open weights" release instead of "open source", which would imo require the training code, dataset and docs. - "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B. - More confusion I see is around expert choice, note that each token and also each layer selects 2 different experts (out of 8). - Mistral-medium
Source: https://twitter.com/karpathy/status/1734251375163511203
New open weights LLM from @MistralAI
params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2
Likely related code: https://github.com/mistralai/megablocks-public
Oddly absent: an over-rehearsed professional release video talking about a revolution in AI.
If people are wondering why there is so much AI activity right around now, it's because the biggest deep learning conference (NeurIPS) is next week.
It's amazing how much hormones control our minds.
I'm wondering how much of a factor the larger token dictionary plays in this progress. Seems like greater token lexicon translates to better learning. What do you think?
Source: How to Talk So Kids Will Listen & Listen So Kids Will Talk - https://www.amazon.com/How-Talk-Kids-Will-Listen/dp/14516638...
Shogi is inferior though to Chess because it takes too long to play with the pieces being returnable.
Example: https://github.com/microsoft/MS-DOS/blob/master/v2.0/source/...