218 karma · joined December 4, 2022
Ultrasensitive low cost magnetometers could usher in a wave of low cost MRIs. The burden would shift from exotic hardware and power electronics to signal processing and computational reconstruction, which scales great.
We're currently running up against the Von Neumann bottleneck in ML. It's a great architecture for general purpose compute, but not suited for efficient operation of neural networks: a ton of energy is spent shuttling weight and activation values between GPU RAM and ALUs. Biological networks appear to be able to process a large amount of information per unit energy by exploiting the static physical structure of the network. A small amount of energy in sparse neural activations can transform a large volume of information stored in the structure and synapse strengths.
Here's some very rough napkin math: assume the brain has 100 billion neurons, each with 1000 synapses, and is sparsely activated at ~5% activation, and processes at 30Hz (gamma wave frequency, roughly). This means every "tick", 5 billion neurons must interact with 5 trillion other neurons. If you tried to implement this in a Von Neumann architecture, even if you only calculate for the sparsely activated neurons, and even if you quantize your weights values to 1 byte, this means processing ~150 TB/s of weight data. That's an insane amount of memory bandwidth. An A100 is 2 TB/s at 300W, but our brain only uses 20W.
My two cents is that the future will involve a hardware architecture that let's us avoid moving these weights around during inference. Whether this will mean a "mortal computer" as Hinton has recently discussed (https://www.youtube.com/watch?v=sghvwkXV3VU) or whether the weights will be loaded and fixed at init time (as with Mythic's or other neuromorphic approaches), time will tell.
No. Biological neural networks run many orders of magnitude more energy efficiently, and learn substantially more data efficiently and continuously.
We've got a lot of catching up to do with nature. In my opinion, backprop on today's compute architecture won't ever compete (I can elaborate if you like) - we need some fundamental changes in our approach.
My money is on something like processing-in-memory enabling inference on gigantic models with rock bottom energy consumption, and rolling out to the masses as a PCIe device. Lots of research on this direction, see https://mythic.ai for an example.
Not useful for cabling, but this could conceivably be used for room temperature SQUIDs [2], paving the way to cheap MRIs.
[1] https://www.engineeringtoolbox.com/young-modulus-d_417.html [2] https://en.wikipedia.org/wiki/SQUID
An exciting step forward if it's reproducible, but not useful commercially. I'd rather see breakthroughs in theoretical understanding; superconductivity in these regimes is still very poorly understood to my knowledge.
[0] https://www.cambridge.org/core/journals/laser-and-particle-b...
But if they're sitting next to another nuke which detonates, the core doesn't even have to go critical for it to release a bunch / most of its energy due to its neighbor's neutrons flux. Similar to how in thermonuclear (fusion) weapons a simple blanket of unenriched uranium can provide a huge boost to the weapon output.
"Compression is comprehension" - Chaitin
This gives you a distribution of predictions, which you can use to produce an average, as well as uncertainty, confidence bands, etc.
The quality of these predictions and ensembles is empirically validated using proper scoring rules (https://en.wikipedia.org/wiki/Scoring_rule), which is a robust way to measure the quality of a probabilistic forecast. Negative-log-likelihood, a ubiquitous loss function in machine learning, is an example of a proper scoring rule (with a minus in front of it).
The science behind forecasting has gotten pretty sophisticated, it's an interesting field.
Your writing itself is great. A lot of writing linked by HN seems too verbose because people try to sound smart. Yours is succinct while engaging. Right length, just technical enough, and interesting. I really enjoyed this, thanks for writing it!
Also:
> Object-orientation is not fashionable right now in gamedev circles
Can you elaborate on this?
However, if you ductaped a simple goal directed reinforcement learning embodied agent onto a large language model, I think you'd approach something resembling animal-like intelligence. It would have motives, and the means to try to achieve them and adapt to it's environment. There's research in using LLMs for "internal dialog" in robots which is super interesting (https://innermonologue.github.io/)
In practice this doesn't happen for covid vaccines, and her personal hypothesis was that many of the adverse reactions (myocarditis, or just being hit hard) were due to the vaccine partially being administered into a vein.
If you can eliminate or reduce the need for armed guards and mountains of red tape, this has the potential to solve many of fission's problems while providing the same benefits (unlimited zero carbon power with dirt cheap fuel).
These may seem like tiny steps forward, but once the genie is out of the bottle it's going to be nuts.
Also this laser tech is ancient, once there's a major economic driver behind it I expect they'll rapidly advance.
> The Mediterranean diet is associated with a reduction in all-cause mortality in observational studies. There is evidence that the Mediterranean diet lowers the risk of heart disease and early death. The American Heart Association and American Diabetes Association recommend the Mediterranean diet as a healthy dietary pattern that may reduce the risk of cardiovascular diseases and type 2 diabetes, respectively.
Based on how transformers are trained to predict the next token, you can view them as powerful compression algorithms [1]. They truly must - and do - understand complex concepts to generate text as they do.
Understanding isn't the full picture for intelligence. Marcus Hutter has argued that AGI requires both compression/comprehension, and being goal directed/having agency. I'd say chatGPT has advanced understanding, but zero agency.