So many details that people gloss over. I have used tensorflow (TF) and it is true that GPUs suck at interference at it. But it's not always the GPUs fault

- TF can't do anything quantized on GPUs. It just switches back to to the CPU/TPU. - TF gets relatively poor utilization of the GPU and tends to not be careful with memory use. - I was able to do certain types of classification hundreds of times faster by seeing what TF was doing it and hand writing it in OCL. Using https://docs.rs/ocl/0.14.1/ocl/. It's a super cool library for rust. Also users should checkout tensorRT https://github.com/NVIDIA/gpu-rest-engine/tree/master/tensor.... It's not super well supported and may go away, but it is fast