Writing an assembler for AMD's GCN ISA would roughly accomplish the same task, except your processor would actually be fast.
Also you don't need the management cores to be fast, it suffices to have lots of GHz+ cores controlling ultra-wide SIMD (as Intel's Xeon Phi has shown).
Arguably, one could get another order of magnitude speedup by implementing XNOR-net (http://arxiv.org/abs/1603.05279) hardware accelerator, but it hasn't been done yet.