The thought of writing Java without IntelliJ makes me want to quick programming. Ironically, Kotlin might be great in a text editor despite being made by Jetbrains.
You might want to make you have parallel builds and the gradle daemon on for the gradle CLI. Android Studio seems to turn these on for you but if you build from the terminal I think they need to be specified in a gradle.properties file in the project root. You can also increase the amount of memory gradle is allowed to use.
As far as I know Kotlin does almost all of its magic at compile time so I am sure it will slow down builds slightly versus plain Java. From what I have seen so far I am more than willing to pay this price.
Here is a good overview with benchmarks from Square
https://docs.google.com/document/d/1ReS3ep-hjxWA8kZi0YqDbEhC...