Meta-Learning: the future for foundation models, and how to improve it

The referenced paper [1] invents a version of evolutionary algorithm where each gene is associated with a target "population ratio" which is modulated upwards or downwards by that gene's performance, and ultimately decides how many offspring that gene will have. The population ratio is not really a ratio. It's just a float that represents the fiction that a particular genome represents a population of identical clones.

A gene is deleted only if its population ratio falls below 0.0001. In this way, the culling decision is softened so that doing poorly on one round of fitness evaluation does not instantly cause the gene to disappear.

On the one hand, this promotes genetic diversity which is a major concern for genetic algorithms. On the other hand, it seems that this would increase computational costs since it becomes much harder for the total population size to decrease.

Absent from the authors' framework is any type of sexual reproduction / crossover. In my experience, sexual reproduction vastly increases the quality of evolved genomes, since high fitness snippets of genetic code rapidly spread throughout the population.

Genetic algorithms are cool, but they take a large population to work well. So it's hard to combine this class of algorithms with deep learning. You might have enough compute for a population of tens of organisms if you can keep your neural networks small.

[1] https://arxiv.org/abs/2103.06435

I'm gonna be honest, I have no clue what this is doing on the front page - there's a whole lot of nothing useful in the article.