In some ways, a single cell is more like an entire artificial neural network.
Real neurons have some amazing properties that individual neurons in ANNs do not. It's still unclear how we can exploit those properties to improve AI.
Real neurons have some amazing properties that individual neurons in ANNs do not. It's still unclear how we can exploit those properties to improve AI.
I would image then, trying to imbue our neural networks with the properties of real neurons is a futile ambition?
In essence an ANN is just a complicated mathematical function that you can tweak to fit the training set. Originally the activation function of a neuron in an ANN resembled real neurons, but later it was found that other, very different, functions are much better, like RLE.
the neuron analogy has been there all along, since ANNs are entirely analogous to "BNNs" - for instance the NIPS conference.
the firing-rate-vs-activation-level difference isn't that big a deal.
Yet LLMs can achive miraculous results with so little. It really begs the question, where is the limit? How smart would a artificial human brain be if it would clock in gigahertz and had the somehow orders of magnitudes more efficient reasoning and detail recall of LLMs?
By lowering the clock speed, power and heat drop exponentially.
I'm just hoping we figure out something logistically better than laminating chips together.
However LLMs are not really tiny compared to real brains, brains FLOP/s have been estimated to be around 1e15 to 1e21, while a single H100 SXM can already do 1e15 BF16 FLOP/s.
The brain also has about 1e15 synapses, each holding the equivalent of about 4.7 bits of information, which would be about 590 TB worth of parameters, equivalent to 294T BF16 parameters (would need 7.4k H100s to hold those, at 80GB per H100).
So in my opinion, it's more of a memory bandwidth problem than a computing power problem, as H100s have comparatively very little memory bandwidth per FLOP than the brain, which holds information directly inside the computing units (synapses).
And I also think that detail recall is a double-edged sword. Memorizing details can certainly be useful, but it can also compete with pattern generalization. If you can perfectly remember the training set (overfit) you don't need to learn to generalize.
Nobody knows how much would be gained by using a more biologically accurate model of a neuron. Much of the low level detail may be irrelevant, but the timing aspect of it, or some equivalent, might be helpful if we want to make ANNs capable of learning realtime pattern recognition and control.
> what appears to be
> Nobody knows how much would be gained by using a more biologically accurate model of a neuron.
Philosophical:
The neuron is an integral part of it's environment... so using models of much more primitive biological neurons ( less "processing" distance between env and self ) might yield something, I believe, but not evolved ones, where they are part of tens of thousands of systems in systems with only a few of them layered like onions. It's a completely different definition of distance and the neural nets are not part of anything.
> Much of the low level detail may be irrelevant, but the timing aspect of it
Tens of thousands of asynchronous contiguous and non-linear events firing at just the right timing in 10s of thousands different places at timings that create something even crazier than perfect synchrony ... That lower level detail is likely responsible for the timing ... it's a bit like why osmotic pressure inside and outside is important and implicit to all the ratios relevant in the different spots of the body/organ/tissue ....
I'm unimpressed by AI and genAI to this day. But I think it's cool. ... it can all be achieved without calling it "AI" at all. Just a bunch of nodes and well sorted and tagged input ... it's social proof laundered awe mostly ... no chance against that marketing machine, so LLM = AI. Every time you "think" it can be more than a dictionary and wikipedia I realize how stupid I am for thinking that briefly when I read the reply to my prompt. Same goes for code. It's holy shit, wow, ah wait, there's that open source code and that free frontend design bit and all the other things, snippets, assets I had already bookmarked somewhere or sometime in the past ...
Where I think the really interesting parts missing are those in real neural circuits do, especially since the connections are more heterogenous, and the dynamics are much more fluid and time-dependent, whereas typical at-scale computer neurons obviously operate at fixed time steps, and don't exactly have bursters etc.
I wrote a very large scale distributed simulations for real neurons (straight from imaging datasets) when I was working in a TMS lab, and a longtime side project I always wanted to get going was some kind of pipeline for spitting out neuromechanical/cable-theoretic RTL tailored to specific morphologies and networking them according to their connectomes. I don't even think I would see any performance improvements, it just feels fun.
For a long time I've wanted to engineer a cell that can pulse like a fluorescent light by computing a taylor series approximation of a sine curve. But to do that you'd have to implement a reliable system and cells already can do this with fairly high accuracy using their internal existing systems.
DNA, the molecule and associated enzymes, is a computer in and of itself. Every dimension - from its hydration to dimensional folding - is a computational manifold.
The cell itself is a system of distributed supercomputers, a machine assembly factory, a dizzying highway and information network, and a universe. Replete with defenses, power factories, intricate and robust signal processing, and self destruct mechanisms.
(That's true for computers (on semiconductor basis) as well, is the behaviour of an addition in the ALU or in the bit pattern to select the instruction? It's both, the ALU alone would exhibit all kind of operations (including unintended/not-meaningful ones), the bit pattern alone could trigger any kind of operation depending on the circuit it occurs in. But if both happen to occur at the same point in spacetime, the physical laws apply and the bit pattern emerges, that humans would call the addition.)
There are a very small number of DNA enzymes produced in labs, but there are no endogenous DNA enzymes (that anybody has found so far). Instead, it seems like life is much more based on RNA enzymes and proteins (the core of the ribosome is an RNA machine, decorated by proteins that enhance its speed and reliability).
I totally see the CheY as an intermediary layer that learned how the changes in attractant must affect mechanical activation.
in short, a continuously running graph that can rewire (instead of a feedforward network) with message passing; it would run training and inference simultaneously (inference would continuously retrain the network). Such a system running on the largest GPU clusters would be an interesting experiment in itself, although I don't think it would match the direct raw performance of today's AI models, and may never actually be better than just using feedforward networks trained by backpropagation, with access to external signals.
I'm also pretty sure people have been, and continue to, experiment with this. The current architecture is by no means the final form of LLMs. It's being tweaked continuously, though you don't hear about it as much anymore (unless it's DeepSeek dropping a bombshell every couple months).
Thing is, despite half the posters on HN and elsewhere arguing otherwise, LLMs are making such a massive, world-transforming progress, that humanity can't keep up with it. A good chunk of the world economy is now focused on AI R&D, and there's still not enough capacity to pick up all the low-hanging fruits found over those few years - there's still too much to gain from pushing the current paradigm.
Every now and then you see someone stumble over one of these low-hanging fruits, and it sparks a mini-revolution. Jev being the most recent example - a simple idea, arguably well-known in ML circles, but it's suddenly a big thing because someone stopped long enough to pick it up. There's many, many more such ideas still waiting for a minute of attention of someone with few million $ worth of GPUs to spare.