PyPI packages don't include the base CUDA install (not sure if you meant to imply that, but it can read that way).
But yes, this is it. If you have a compiled library based on CUDA that results in a pretty large binary, and you need one for each version of the CUDA API * Python version * OS, that adds up quickly.