Further, while all these tricks are amazing, they are not currently of much use to fungible data scientists at FAANG and in the greater Fortune 500 because they haven't been added to any of the major frameworks in a manner that is braindead/automagic. Further, most of these are but one GPU generation away from general availability if they become popular like FP16/32 mixed precision did.
That said, the acquisition of DeePhi by Xilinx ought to be a sufficiently strong signal to add them to said frameworks, no?
https://www.xilinx.com/news/press/2018/xilinx-announces-the-...
Finally, my gut instincts (based on the monotonically decreasing speech recognition quality of my increasingly powerful Android phones over time, and the bizarre spelling errors it now makes such as "tech kneeq" as one example) is that one cannot rate the quality of these techniques from a single metric, and it seems like the decision to foist them on users may be based on one. #ItsAdversariesAllTheWayDown