935 karma · joined March 6, 2023
You can write your own allocator in C. You don't have to use malloc.
“Pretraining” is not a correlate of the learning paradigms, it is a correlate of the “fine-tuning” process.
Also LLM pretraining is unsupervised. Dwarkesh is wrong.
Never used it, can't vouch for it. But it's under 100 MB. The model it's based on, gte-tiny, is only 46 MB.
* Energy physics puts an upper bound on material wealth
* The disparity between notional and material wealth is large and growing
* Notional wealth figures are largely speculative / fictitious
Claims that could be true (if empirically verified):
* Material wealth is decreasing
* Energy is decreasing
* The monetary system will collapse
Claims that have been implied but not demonstrated or argued:
* There is a causal link between decreasing energy supplies and monetary system collapse
Overall it smells like 2004-era peak oil doomerism. I’m not saying it’s wrong, it could just be early. Intrigued but not convinced.
That’s what “AI alignment” is. Doesn’t seem to be hurting Western models.
Ruling elites that consider the interests of the majority? Novel idea.
The examples in the playground are neat but I don’t see any of the Python bread and butter usage. I see C with Python syntax.
I’d like to see Python syntax solving a problem in a pythonic way, albeit with type annotations. And then I’d like to see the compiled version run 30x-100x faster. Is that ultimately the promise of this project?
# - the unsafe module allows C-level direct memory access to pointers and unsafe arrays
# - @struct maps directly to C structs
# - most users will never have to deal with this directly: using the unsafe` module is the equivalent of writing C extensions or using Cython
That said, the authors are saving this for future work. Fine-tuning is cheaper, easier, faster to validate.
>Switching to a new architecture at pretraining time has a high cost, but there are reasons we might want this (besides the better scaling behavior). The main benefit is that the model can learn to organize its memory from scratch, and once we’ve already “allocated” this high-capacity memory pool, there’s a clearer path to learning on multiple tasks and corpora over time.
This means you could "fine-tune" the model on your custom corpus at ingestion time, without having to actually train via backprop. Your corpus would be compressed into model-readable memory that updates model behavior. Then different memory units could be swapped in and out, like programs on a floppy disk. I can see this concept being especially useful for robotics.