The kind of GPGPU code where the language is a thin architecture agnostic layer over the opcodes is how most GPU code (eg shaders used in graphics applications) are implemented. This is also how OpenCL, Vulkan Compute etc do their thing. This approach requires a lot of boilerplate and babysitting, but works well for relatively short bits of code.

CUDA is much higher level. It's roughly on par with a "C++ is C with classes" level in terms of language capability. This makes it much easier to develop complex applications. The C compatibility means that you can reuse the exact same code between CPU and GPU in many cases. It eliminates a lot of boilerplate, since you don't need to manage your data in as much detail (eg, while you still have to make sure your pointers are valid for the GPU, the code for uploading the function arguments is generated by the CUDA compiler).

The value add that makes CUDA especially strong is all the first party libraries which have been carefully optimized and have widespread and proven long term support.