Yes, stuff like https://developer.android.com/ndk/guides/neuralnetworks
Decades ago, compiler developers solved dealing with multiple CPU architectures, by creating languages that succinctly expresses their common traits: Doing arithmetical-logical operations and accessing memory. Languages like C abstracted us from CPU registers, stacks. Optimization passes like vectorization abstracted us from different ISA extensions.
Now it feels it's the same issue all over again. Is it really hard to simply express: `x' = W*x + b` and let the compiler target the CPU/GPU/NPU as needed?