The author even shouts out esbuild as being "a little more stable", which Vite uses under the hood.
But the core package is ~27k SLOC with upwards of 40 dependencies. Thats an indirect, but poignant statement illustrating how much work goes into solving just a tooling problem. And while it may hide some of the overhead in its own abstraction, which it does a fantastic job of, the overhead is still there and it can still break in arcane ways.
I think the spirit of the question of "why do we even need Vite in the first place?" is whats really being explored in the original post.
I certainly don't shy away from build tools at all, but I do often try to start projects without them just as an exercise to see if they really ever end up being needed. Especially when its so easy to drop-in something like Vite after the fact if its necessary and/or clearly adds an outsized return on investment.