Edit: maybe I need to investigate this again because it isn't working for me. In a usual month I downloaded pytorch through UV maybe 100 times.
Not true, each virtual environment has its own physical copy on disk.
Since creating venvs is usually per project you have many downloads / copies.