No, that's expected. The program you wrote might be minimalistic, but there is still a lot of code under the hood for the runtime.
If you compile a program that's twice as complex, you should still get roughly the same build time.
FWIW The compiler builds in about 30 seconds. And it's a really complex piece of software.
There is a cache that was previously cold. Subsequent runs should be faster. However compile times are longer than other languages like Go.
In .cache/crystal/? 2 seconds is with that. If I delete it, the next build takes 3.