HNHacker News
TopNewBestAskShowJobs

dontwearitout

218 karma · joined December 4, 2022

submissionscomments
dontwearitout··on Volkswagen’s ID. 2ALL Is a $26,000 EV Hatchback You’d Want to Drive
For electric vehicles it's not that much more complex compared to traditional ICE vehicles.
dontwearitout··on Viable superconducting material created in Rochester lab
https://en.wikipedia.org/wiki/SQUID

Ultrasensitive low cost magnetometers could usher in a wave of low cost MRIs. The burden would shift from exotic hardware and power electronics to signal processing and computational reconstruction, which scales great.

dontwearitout··on Large language models are having their Stable Diffusion moment
My opinion is heavily shaped by this interview (5-9 minutes is a good summary): https://www.youtube.com/watch?v=0PAiQ1jTN5k.

We're currently running up against the Von Neumann bottleneck in ML. It's a great architecture for general purpose compute, but not suited for efficient operation of neural networks: a ton of energy is spent shuttling weight and activation values between GPU RAM and ALUs. Biological networks appear to be able to process a large amount of information per unit energy by exploiting the static physical structure of the network. A small amount of energy in sparse neural activations can transform a large volume of information stored in the structure and synapse strengths.

Here's some very rough napkin math: assume the brain has 100 billion neurons, each with 1000 synapses, and is sparsely activated at ~5% activation, and processes at 30Hz (gamma wave frequency, roughly). This means every "tick", 5 billion neurons must interact with 5 trillion other neurons. If you tried to implement this in a Von Neumann architecture, even if you only calculate for the sparsely activated neurons, and even if you quantize your weights values to 1 byte, this means processing ~150 TB/s of weight data. That's an insane amount of memory bandwidth. An A100 is 2 TB/s at 300W, but our brain only uses 20W.

My two cents is that the future will involve a hardware architecture that let's us avoid moving these weights around during inference. Whether this will mean a "mortal computer" as Hinton has recently discussed (https://www.youtube.com/watch?v=sghvwkXV3VU) or whether the weights will be loaded and fixed at init time (as with Mythic's or other neuromorphic approaches), time will tell.

dontwearitout··on Large language models are having their Stable Diffusion moment
Very exciting stuff, thanks for sharing this. They don't have a ton of open positions, but I'll be keeping an eye on the careers page.
dontwearitout··on Large language models are having their Stable Diffusion moment
>Artificial neural networks with backprop is all we need for AI

No. Biological neural networks run many orders of magnitude more energy efficiently, and learn substantially more data efficiently and continuously.

We've got a lot of catching up to do with nature. In my opinion, backprop on today's compute architecture won't ever compete (I can elaborate if you like) - we need some fundamental changes in our approach.

dontwearitout··on Large language models are having their Stable Diffusion moment
I think the answer to this is that we don't anything that much better than GPUs to justify dedicated hardware - but I see it as inevitable. (Edit: I'm not counting TPUs, since they're not for sale).

My money is on something like processing-in-memory enabling inference on gigantic models with rock bottom energy consumption, and rolling out to the masses as a PCIe device. Lots of research on this direction, see https://mythic.ai for an example.

dontwearitout··on Room-temperature superconductor discovery meets with resistance
After a bit of googling, the yield strength of some common materials is within striking distance of 1GPa (many steels are 0.25 GPa, silicon carbide is 3.4 GPa [1]). So perhaps it could be possible to embed this superconductor in a highly stressed supporting structure to provide the pressure.

Not useful for cabling, but this could conceivably be used for room temperature SQUIDs [2], paving the way to cheap MRIs.

[1] https://www.engineeringtoolbox.com/young-modulus-d_417.html [2] https://en.wikipedia.org/wiki/SQUID

dontwearitout··on Room-temperature superconductor discovery meets with resistance
Room temperature, but 1GPa pressure (or 145037 psi).

An exciting step forward if it's reproducible, but not useful commercially. I'd rather see breakthroughs in theoretical understanding; superconductivity in these regimes is still very poorly understood to my knowledge.

dontwearitout··on US-Japan team hails H2-boron plasma fusion breakthrough
I read some interesting work about picosecond laser driven pB11 fusion [0]. There doesn't seem to be a huge amount of research into picosecond lasers, but it strikes me as the logical step past NIF's work. What are your thoughts on this?

[0] https://www.cambridge.org/core/journals/laser-and-particle-b...

dontwearitout··on In neutron stars, astrophysicists see a form of matter like none other
Forces due to a dipole effect (like magnetism) fall off like 1/r^3, while the strong nuclear force is exp(-x). There might be some analogy between the two in some sort of unification theory, but the power law suggests that it's fundamentally an unrelated phenomenon to me.
dontwearitout··on Declining sperm count: Much more than you wanted to know
I agree - but reproductive capacity aside, this indicates some very concerning trends in men's health which we shouldn't brush off.
dontwearitout··on Astronomers uncover a one-in-ten-billion binary star system
It took me a minute to grok the pictures on that wiki page, but I love this! Super creative solution
dontwearitout··on An obituary for the man who saved North Carolina from Nuclear Disaster
I'm not an expert, but my understanding is nukes are designed to not detonate if they are damaged or destroyed by a conventional explosion.

But if they're sitting next to another nuke which detonates, the core doesn't even have to go critical for it to release a bunch / most of its energy due to its neighbor's neutrons flux. Similar to how in thermonuclear (fusion) weapons a simple blanket of unenriched uranium can provide a huge boost to the weapon output.

dontwearitout··on Light from an ionized state of helium in a distant galaxy
Thank you for sharing this!
dontwearitout··on How to get 1.5 TFlops of FP32 performance on a single M1 CPU core
One issue with this is that many ML tasks are more memory bandwidth bound than compute bound. A forward pass through a neural network is a bunch of simple operations on gigabytes of data, whereas with protein folding there's a much higher ratio of compute to data.
dontwearitout··on VALL-E: Neural codec language models are zero-shot text to speech synthesizers
Absolutely. These transformer models are trained to autoregressively predict the next token (text, or now audio), and so can be viewed as powerful compression schemes by combining with something like Huffman coding (see https://ml.berkeley.edu/blog/posts/dalle2/ for a lengthy but great read on how language models can be viewed as sophisticated lossless compression algorithms). The log-likelihood loss function used to train these models can quite directly be interpreted as the average bit-per-token needed to compress the dataset.

"Compression is comprehension" - Chaitin

dontwearitout··on Incoming potentially catastrophic storm for SF area
They are running ensembles of models to predict the forecast. Chaotic systems are challenging to model, and models aren't perfect, so a collection of models is used (either different initial conditions or different versions of the model, as per https://forecast.weather.gov/glossary.php?word=Ensemble).

This gives you a distribution of predictions, which you can use to produce an average, as well as uncertainty, confidence bands, etc.

The quality of these predictions and ensembles is empirically validated using proper scoring rules (https://en.wikipedia.org/wiki/Scoring_rule), which is a robust way to measure the quality of a probabilistic forecast. Negative-log-likelihood, a ubiquitous loss function in machine learning, is an example of a proper scoring rule (with a minus in front of it).

The science behind forecasting has gotten pretty sophisticated, it's an interesting field.

dontwearitout··on Safari's date-picker is the cause of 1/3 of our customer support issues
Sounds like a job for a LLM browser plugin :P
dontwearitout··on I still use Flash
Flash was my 2nd programming language (after VB6) back when it was still Macromedia. I had a lot of fun making really crappy minigames and learning the basics of coding. This article brought some good nostalgia and plenty of astonishment at the lengths you went to.

Your writing itself is great. A lot of writing linked by HN seems too verbose because people try to sound smart. Yours is succinct while engaging. Right length, just technical enough, and interesting. I really enjoyed this, thanks for writing it!

Also:

> Object-orientation is not fashionable right now in gamedev circles

Can you elaborate on this?

dontwearitout··on Building a Virtual Machine Inside ChatGPT
Not in any real sense - it just reacts to the prompt, and has no choice in how.

However, if you ductaped a simple goal directed reinforcement learning embodied agent onto a large language model, I think you'd approach something resembling animal-like intelligence. It would have motives, and the means to try to achieve them and adapt to it's environment. There's research in using LLMs for "internal dialog" in robots which is super interesting (https://innermonologue.github.io/)

dontwearitout··on Autopsy-based characterization of myocarditis after anti-SARS-CoV-2-vaccination
A nurse friend of mine raised this as a concern. The protocol for intramuscular injections used to be (and in some places still is) to draw the plunger back a bit first to verify that you're in muscle, and haven't hit a small vein. If you pull blood, you find new spot.

In practice this doesn't happen for covid vaccines, and her personal hypothesis was that many of the adverse reactions (myocarditis, or just being hit hard) were due to the vaccine partially being administered into a vein.

dontwearitout··on Autopsy-based characterization of myocarditis after anti-SARS-CoV-2-vaccination
I agree, it'll probably be scrubbed from the frontpage shortly. It's sad that this is the case, especially since it's published in a scientific journal.
dontwearitout··on US Department of Energy: Fusion Ignition Achieved
Fission's amazing potentially is nerfed by three main things: proliferation risk, meltdown risk, and waste handling. These are all solved problems but dramatically raise the cost (you need armed guards, and the reactor has to be built to withstand a 747 strike, etc etc). The fuel cost is a very small fraction of the price of nuclear.

If you can eliminate or reduce the need for armed guards and mountains of red tape, this has the potential to solve many of fission's problems while providing the same benefits (unlimited zero carbon power with dirt cheap fuel).

dontwearitout··on Fusion energy breakthrough by Livermore Lab
I like to remind people now and again that humankind's very first attempt at inertial confinement fusion was wildly, ridiculously, terrifyingly successful: https://en.wikipedia.org/wiki/Ivy_Mike.

These may seem like tiny steps forward, but once the genie is out of the bottle it's going to be nuts.

Also this laser tech is ancient, once there's a major economic driver behind it I expect they'll rapidly advance.

dontwearitout··on What causes Alzheimer's? Scientists are rethinking the answer
Individual needs vary, but the Mediterranean diet is a great place to start: https://en.wikipedia.org/wiki/Mediterranean_diet

> The Mediterranean diet is associated with a reduction in all-cause mortality in observational studies. There is evidence that the Mediterranean diet lowers the risk of heart disease and early death. The American Heart Association and American Diabetes Association recommend the Mediterranean diet as a healthy dietary pattern that may reduce the risk of cardiovascular diseases and type 2 diabetes, respectively.

dontwearitout··on Drowning in AI Generated Garbage: the silent war we are fighting
It's made me consider my habits, and how much time I spend consuming internet content or comment threads which may (even today) be completely fabricated or autogenerated. I'm hoping I can use this inflection point as motivation to unplug a bit and spend more time enjoying being human and enjoying the real world.
dontwearitout··on Building a Virtual Machine Inside ChatGPT
Strongly disagree. Mathematician Gregory Chaitin said "compression is comprehension", and I think this is exactly correct: something is "more understood" if it can be internally represented or externally reproduced with less information.

Based on how transformers are trained to predict the next token, you can view them as powerful compression algorithms [1]. They truly must - and do - understand complex concepts to generate text as they do.

Understanding isn't the full picture for intelligence. Marcus Hutter has argued that AGI requires both compression/comprehension, and being goal directed/having agency. I'd say chatGPT has advanced understanding, but zero agency.

[1] https://ml.berkeley.edu/blog/posts/dalle2/

← PreviousPage 3 of 3