We....do use kernels and kernel generation in the ML field. Every day. I'm confused as to a few of the points made here, unless the first sentence is meaning '...doesn't use that many hand-written kernels'.

All of the convolutions we have run on kernels, some pre-built and customized/chosen from a list based on performance, and some dynamically generated. PyTorch 2.0 for example decomposes and fuses operations, then uses OpenAI's Triton to dynamically generate a custom fused kernel that tends to be very efficient.

There are still hand-written kernels, even, Flash-Attention and the Memory-Efficient attention papers both caused huge leaps forward because they manually went through a lot of the inefficiencies of naive matrix multiplies for attention w.r.t. the hardware design and optimized it quite a lot.

I think generalized kernel generation though may have more life in it than you might suspect! It is a fascinating field and I do not know nearly enough about it. I hope someday to be able to write my own Triton kernels/get to know how it integrates as a dynamic compiler for PyTorch code. We certainly live in wild times. Crazy indeed.