It's funny that I'm currently going in the completely opposite direction of using JavaScript to build AI apps (with tensorflow.js). I'm a frontend dev of course.
I think using one language for everything does have the benefit of reducing context switching.
It's about picking your comfort language for higher productivity, while making sacrifices somewhere like performance and less idiomatic.