I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?
We use Kubernetes to trigger rolling restarts with new images when we release a change to the base image, and so far it's been painless, but a lot of work went into it. We use Gitlab, but any CI/CD should allow you to do it.