I'll add some highlights for those who don't have the time - Flash Attention 2 kernels, Paged Attention kernels, PyTorch, vLLM, transformers, ONNX runtime support, OpenAI triton support in v3, tensorflow, JAX, OpenXLA.

Also, while this wasn't in the presentation, CUDA can be translated to HIP using automated tools. and - CuPBoPAMD is a CUDA translator that translates CUDA programs at NVVM IR level to HIP-compatible IR that can run on AMD GPUs (https://dl.acm.org/doi/pdf/10.1145/3624062.3624185)