If your vision is weird or niche, building a toolset around that from day one can be way more efficient in the long run
The best tool is the one you know, even if your vision is weird or niche. At the end of the day, you can always ignore all the bits and pieces unity gives to you and just write custom logic in your MonoBehaviour scripts and use it as a platform toolkit, input handler, content pipeline, scriptable editor, and renderer. There's a lot to be said for the features you get from that especially in the long term as you said.