Serious vim users don’t try to create a kitchen sink IDE. They start with a vanilla base and with their way up, including just the features that are important to them.
Starting with something like LazyVim is fine too. Then you get a rich set of plugins that have been configured for you. Lovely! And now just be thoughtful about what you later on top.