(note that rusticl + zink does deal with it _partially_ to some extent nowadays)
+ Vulkan memory management doesn't expose unified address space primitives
Why would you want unified address space? At least in my experience, it’s often too slow to be useful. DMA transfers (CopyResource in D3D11, copy command queue in D3D12, transfer queue in VK) are implemented by dedicated hardware inside GPUs, and are way more efficient.
OpenCL is stricter with the results of floating point operations, and makes different assumptions with respect to memory aliasing. Whether or not this is important the AI domain I don't know.
> Why would you want a unified address space?
A unified address space doesn't always imply that the memory can be accessed from anywhere (although that might also be supported with some memory allocation mechanisms), and you still may have to copy between host and device memory. But it makes it much easier to have pointers in your GPU kernels, instead of having to deal with objects like OpenCL buffers.
Mac APU I guess. Or Jetson/Tegra kind of things.
CPU-style memory access in GPU kernels technically possible (CUDA did it) but unfortunately rather hard. The feature requires hardware support inside GPUs (need pointers, they need to be 64-bits, need 64-bit integer arithmetic instructions), and therefore not going to work on many current GPUs. It becomes harder to compile and optimize GPU-running code. On devices without physically unified memory, the performance of these kernels gonna suck.
Luckily, none of that is needed to implement D3D11 or Vulkan backend for PyTorch. PyTorch is not a general purpose GPGPU runtime, it’s merely a high-level library which manipulates tensors and implements a few BLAS routines operating on them. It’s easy to allocate a GPU resource for each tensor being manipulated.