Self-Assembling Artificial Neural Networks Through Neural Developmental Programs
arxiv.org
arxiv.org
wouldn't evolutionary algorithms be better suited for this ? evolution is how we got here after all ? perhaps different constraints on the 'state space' can 'guide' the whole process...
There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on.
Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.
These problems are often enough to prohibit GA use.
i was naively thinking that architecture should be evolved, while weights should be computed just as it is being done now i.e. via back-propogation etc.
> Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.
This is true for the living as well.
Is it? The “easy” tasks for neural nets to beat humans at are things that are harder for humans to do than for humans to provide volumes of high-quality training data by way of which neural nets can be trained.
It’s not clear to ne that designing neural nets really fits that.
That kind of knowledge still needs a human engineer to be leveraged, but even now it seems it's more because nobody has built a harness for the LLM to be able to put together neural network architectures. And, there are already automatic constraint checkers for NN architectures, so even if an LLM hallucinates something wrong, constraint checks could be used to automatically call its attention to some design flaws.
A NN could design a better wing for a bird, but a human can design a jet engine. While the NN could eventually get to the jet engine, all the computation might be better used to train a human design architecture on more data/more epochs/ more complex optimization
> Biological nervous systems are created in a fundamentally different way than current artificial neural networks. Despite its impressive results in a variety of different domains, deep learning often requires considerable engineering effort to design high-performing neural architectures. By contrast, biological nervous systems are grown through a dynamic self-organizing process. In this paper, we take initial steps toward neural networks that grow through a developmental process that mirrors key properties of embryonic development in biological organisms. The growth process is guided by another neural network, which we call a Neural Developmental Program (N DP) and which operates through local communication alone. We investigate the role of neural growth on different machine learning benchmarks and different optimization methods (evolutionary training, online RL, offline RL, and supervised learning). Additionally, we highlight future research directions and opportunities enabled by having self-organization driving the growth of neural networks.
Its development is governed by a DNA algorithm.
Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems.
Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.
Not critiquing the tech. Critiquing some of its rationalization.
You could pretty easily raise one twin as an imbecile and the other as competent, purely by the means of changing their sensory inputs during the phase of brain development. If you had no scruples, of course.
Its development is governed by a DNA algorithm.
Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems.
Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.