One thing that helps me managing Python environments in VS Code: I start VSCode from the command line with my virtual environment activated ("code -n .").
With this, I rarely have issues with VSCode picking up my environment. Hope that is of some use to you.