1,728 karma · joined January 15, 2015
But for a high performance compiler, manually managed arena allocation will beat out a generational GC.
No doubt our world is far from perfect, but does he honestly believe destruction of technology would somehow magically improve the situation, when we know from history that our technological past was full of just as much, if not more, unnecessary suffering than the present?
Vigilance is necessary, yes. Reducing our reliance on fickle technology is valuable, yes. Practicing mindfullness, in our use of technology as in other aspects of everyday life, is beneficial, yes. But, no, destruction is not merited.
Some of the most successful ML researchers have read several decades of research papers. It's not uncommon to see references to papers from the 70s or 90s.
Edit: No doubt that the relevant parts of stats, random matrix theory, and ML is a newer field than Physics or Chemistry, though.
Also, yes: larger space telescopes. And eventually,manned missions to Mars.
I don't think Hyena can be run in RNN mode, though. But it _might_ scale better than the LRU model (second paper).
May I ask what you are using it for?
- "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf
- "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf
It's unclear (at least to me) if any of these scale as well as transformers at this point.
More recently, there are models like RWKV that can run in both parallel (GPT-like) mode for training and serial (RNN-like) mode for inference.
But transformers always output a probability distribution at each position in the context.
To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probability distribution for each possible integer, with the array shifted left by one place to enable prediction.
I've been out of the semiconductor materials and devices game for a while, but I wouldn't be so sure. Just because these materials have a lot of papers about them, doesn't mean they're the future. They get a lot of papers written about them because they're relatively easy to study and because governments are funding research on 2D materials (though I can't really say why they receive so much funding. It's completely disproportional to their promise.)
Even if you could manufacture wafer scale 2D material transistors with high yield (which is a huge if), what does it buy you? Based on my, admittedly out of date, knowledge of 2D semiconductor devices, they're no panacea. Contact resistance is even more of a problem than it is for silicon at leading edge node scales. But even if it could match, or maybe even slightly beat the performance of silicon (another huge if), again, what does it buy you? One extra node, if you're lucky?
No, if my money had to be on a CMOS successor, it would be on nanoscale vacuum switches. Basically a vacuum tube scaled down to modern VLSI dimensions, at which atmospheric pressure is sufficiently low pressure to operate. Their biggest problems are 1) the lack of a complementary switch (so no CMOS equivalent, at least not without MEMS which doesn't scale), and 2) device longevity, which is likely solvable. These devices are far more manufacurable than 2D semiconductor material transistors. All of the process steps already exist, they can be stacked, unlike semiconductors, which must be single crystalline for good performance and therefore cannot be grown on insulators, device characteristics have a much higher ceiling thanks to the full vacuum barrier, and contact resistance is far less of a problem, because all of the junctions are between metals.
My second runner up would be room temperature superconductor switches, but we don't have that technology yet.
Edit: looks like there's now confirmation that running it on a 10GB VM slows inference down massively, so looks like the only thing strange is the memory usage reading on some systems.
And there's several research papers about other architectures. For example: https://arxiv.org/abs/2205.05853
Another recent paper on efficient architecture for long context lengths: https://arxiv.org/abs/2302.10866