You're right, --user seems the default behaviour, I did not realize that. Maybe because it's different in Fedora that I also use? Or maybe I was just plain wrong :)
Anyway, I think this does not change the fact that it's a bad idea to populate your ~/.local/lib with python libs, since it may break some python programs installed via apt for your user, or was that wrong too?