Are there any alternatives to gradient based learning that could make this less useful? Is there another type of compute unit that is the next evolution of CPU -> GPU -> ?
1) All AI is based in search (esp. non-convex, where heuristics are insufficient to provide a global convex solution), and thus is inevitably implemented using iteration, driven locally by gradient-pursuit and globally by... ways to efficiently gather information to optimize the loss function that measures how well that info gain is being refined and exploited.
2) Search that is inherently non-convex and inefficient requires as much compute power as possible, i.e. using supercomputers.
3) All supercomputer-based solutions to non-convex problems are implemented iteratively, where results are improved not using closed-form math or complete info, but by incremental optimization of partial results that aggregate with the iterations, like repeated stochastic gradient descent that creates and enhances 'resonant' clusters of 'neurons'.
4) The only form of supercomputing that has proven to scale up at anywhere near indefinitely is data-parallelism (a dataflow-specific form of SIMD) -- where the search space is spread as evenly (and naively) as possible across as many processing elements as possible.
5) Vector processing hardware like GPUs implement data-parallelism as well as any HPC architecture yet devised.
Thus, I believe that AI is stuck with GPUs, or equivalent meshes of vector processors, indefinitely.
Quantum computers are not magic and they are by no means general-purpose, but if your search mechanism matches the kind of search quantum computers are good at, they can greatly exceed the speed of GPUs.
If so, then no, using randomness to disrupt search only resets the search to continue at a different (random) spot in the global search space. That kind of approach certainly can escape local minima, but it doesn't diminish the amount of total time/effort required to do (non-convex) global search.
I think the long history of our attempting to use randomness to improve search has shown that all naive approaches inevitably fail unless the number of processors available can be scaled up to 'equal' the number of decisions required to explore the entire search space (as quantum computing seeks to do). But maybe the formal resolution of that question will come only after we prove P != NP.
The lottery tickets are random but they are basically - as I understand - smaller graphs that have the same strength as a larger graph.
It's tough to come up with a new architecture that can have an advantage on current and future models, at least from a peak perf point of view, from a perf/watt for example instead the scaled-up Apple GPUs seem to show new interesting properties. But the Graphcore architecture is quite interesting, being able to act somehow both as a SIMD machine and a task-parallel machine.