Orders-of-magnitude improvements in compute efficiency are needed to become a practical replacement for backprop… but those improvements are coming.

As a mixture, could activation-space search produce useful teaching targets for backprop?

Zeroth-order search would discover candidates, first-order learning would consolidate them. The potentially valuable step is converting a sparse judgment into a reusable training target.

This also changes the relevance of convexity.