Your guess is correct. Pip does keep a cache of packages around. It's easy to fix by setting the `--no-cache-dir` switch when installing:

`pip install --no-cache-dir -r requirements.txt`