A gene is deleted only if its population ratio falls below 0.0001. In this way, the culling decision is softened so that doing poorly on one round of fitness evaluation does not instantly cause the gene to disappear.
On the one hand, this promotes genetic diversity which is a major concern for genetic algorithms. On the other hand, it seems that this would increase computational costs since it becomes much harder for the total population size to decrease.
Absent from the authors' framework is any type of sexual reproduction / crossover. In my experience, sexual reproduction vastly increases the quality of evolved genomes, since high fitness snippets of genetic code rapidly spread throughout the population.
Genetic algorithms are cool, but they take a large population to work well. So it's hard to combine this class of algorithms with deep learning. You might have enough compute for a population of tens of organisms if you can keep your neural networks small.