Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two.
wouldn't evolutionary algorithms be better suited for this ? evolution is how we got here after all ? perhaps different constraints on the 'state space' can 'guide' the whole process...
There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on.
Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.
These problems are often enough to prohibit GA use.
> Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.
This is true for the living as well.
i was naively thinking that architecture should be evolved, while weights should be computed just as it is being done now i.e. via back-propogation etc.
Is it? The “easy” tasks for neural nets to beat humans at are things that are harder for humans to do than for humans to provide volumes of high-quality training data by way of which neural nets can be trained.
It’s not clear to ne that designing neural nets really fits that.
That kind of knowledge still needs a human engineer to be leveraged, but even now it seems it's more because nobody has built a harness for the LLM to be able to put together neural network architectures. And, there are already automatic constraint checkers for NN architectures, so even if an LLM hallucinates something wrong, constraint checks could be used to automatically call its attention to some design flaws.
A NN could design a better wing for a bird, but a human can design a jet engine. While the NN could eventually get to the jet engine, all the computation might be better used to train a human design architecture on more data/more epochs/ more complex optimization