Do you still have latency issues in Julia 1.6? The latency improvements in the last 3 versions of julia have been so significant that I do not really notice it anymore. Supposedly there are additional speedups planned for 1.7.
Beside this, if you only infrequently install/update package, you can use PackageCompiler.jl. I use it for PyPlot.jl (based on matplotlib), DataFrames.jl, ... and plotting some data quasi instantaneous as it is in python (even the very first time in a session).