Half agreed, but, there are some mitigating things to consider.
One is that native code is really verbose compared to bytecode, and jlink optimizes bytecode to shrink it when it's modularized (collapses the string tables and other things). For tiny hello world apps native-image can produce smaller downloads. It doesn't take long before a jlinked JVM is smaller, because the JIT compiler is kind of "decompressing" native code on the fly. A minimal jlinked JVM can be small-ish, like 20mb or less, and if you compile a custom JVM with configure flags for size it can be even smaller still. The smallest jlinked hello world I managed to get when I tried this was a 7mb download. Not small for a program that only prints something, but, obviously, most of that is one time cost.
Another is that jlink+modules is way easier to use and configure than native-image. Tree shaking breaks stuff all over the place. You end up needing complex config files, agent-based config auto generation and/or lots of runtime testing to ensure you got everything. The point of "reliable configuration" is to avoid all this by building a less aggressive form of tree shaking into the language itself.
Quite a few libraries support modules at this point, but you may not actually have noticed. It depends on your build tool and things.
One thing that's slowed it down is that a lot of widely used libraries that sit at the bottom of the dependency stack aren't actually well maintained. And, modules can only depend on other modules. SLF4J, Guava etc have been slowing things down a lot.
My feeling is that the module system is kind of underrated and waiting for better tooling to be fully exploited. Quite a few parts of Java are like this. Like, it took quite some years for the ecosystem to start using annotation processors to full advantage, and some popular libraries still have it as a pending feature (like PicoCLI).