Artist time costs more money than software licenses do, and as the open source software isn't generally good enough for their needs, using better off-the-shelf software that have fairly decent and flexible plugin APIs is good enough for them.
A lot of the code VFX studios have is proprietary (as in unique to the individual studio who wrote it) plugins for host apps like Maya, Katana and Nuke, and shaders/integrators for renderers like PRMan and Arnold.
There are open source projects that are used for glue in the VFX industry:
https://github.com/alembic/alembic https://github.com/PixarAnimationStudios/OpenSubdiv https://github.com/imageworks/OpenShadingLanguage https://github.com/ImageEngine/gaffer https://github.com/dreamworksanimation/openvdb https://github.com/openexr/openexr
But these have almost exclusively been internal proprietary projects started within studios that have been open-sourced, to help transfer assets between studios, and are mostly open file formats for images, volumes and geometry, as well as other intermediate tooling.