Can you elaborate on what makes containerized CUDA so difficult?
I would guess the fact that its a highly proprietary design that is designed to specifically scale to massive amounts of parallel threads on specific physical hardware devices
the CUDA library versions are very coupled to the hosts' CUDA driver versions and the container system itself (Docker) needs special code to link the two
Can't you just do a host mount/bind mount of those specific libs though?
My last project where we needed CUDA, we ended running our GPU workloads directly on a dedicated machine loaded up with GPUs. The rest of the system was containerized. Though, we were just a small team of software engineers working on an R&D project - we didn't have any k8s wizards on our team.