Yes, NVIDIA GPUs can indeed read directly from main memory.
It's called Heterogeneous Memory Management.
https://developer.nvidia.com/blog/simplifying-gpu-applicatio...
This would allow you to mmap() the weights file into CPU memory, and pass that CPU pointer to the GPU and allow it to fault the data in on-demand.
The GPU will have less bandwidth accessing this memory than if you copied it onto the GPU though, as it will need to take a trip over PCIe to read it. But it would eliminate the need to manually upload data to the GPU and the synchronisation involved with that.