That said, there's a difference between making an engine for a game like Minecraft vs. the kind of engine needed for Limit Theory.
At the end of the day, it's the same stories and questions about core competency present in almost every business. If you want to build a shelf, and you have to start by making the tools that will cut the tree that will become the shelf... it's going to be a while until you see a shelf, and in the process you'll probably be producing more useful tools than shelves.
Unity is popular for the same reasons AWS is popular: It allows you to focus your time on what you excel at, instead of spending it reinventing various wheels. Some businesses (and games) can succeed despite reinventing those wheels, not necessarily because of it.
And in some extremely rare cases, "what you excel at" is both building the engine and making an excellent game out of it (Present company included; I'd recognize your name anywhere). Stardew Valley and The Witness are the only two examples I can give off-hand of significant, outstanding, indie games that were released with a custom engine. And I would maintain that in both cases, it wasn't necessary to create the engine. In SDV's case it's even questionable if it actually helped at all.
Every other example I could come up predates the days of high quality engines being readily available to sub-$100k budget games.
Edit: I'll add Psychonauts to that list; once again, a decision of questionable gain that the author AIUI regretted.
Minecraft, or anything anyone has even heard of, is very much an exception to the norm.
Having investigated every 2D and 3D engine for the web (it's a nice UI, not a game), it seems none of the existing options scratch my itch: SVG and its huge complex object graph grind to a halt burning CPU after your first few thousand nodes, while game engines that can often cope with huge entity counts just continually burn CPU and apparently have no concept of 'idle'.
So the itch is there: just a little <canvas>, a few wrapper objects, and I'm sure I'll be done in no time. Just gotta reimplement D3's TopoJSON renderer, that's all.
Of course it's a lie, but at this stage I'm old enough to realize that :) I think the only way many people (just like me) in IT learn this is the hard way: be getting burnt too many times.
This is not OK when you have made promises to people who expect a product. This guy took advantage of those people, and it's incredibly sad. I don't know how they kept on doing this for 6 years without reassessing if their approach was really the correct approach. I can't fathom that in 6 years, not one person raised their hand and asked "why the hell are we writing a game engine?".
I have seen projects in industry that deliver a button of a different color for more.
At that point, aren’t you almost writing your own engine anyway? You’d need to design the abstractions, interface with the game engine, and consider how those two things impact performance. That’s a big chunk of what a game engine is supposed to do for you.
---
Honestly though, I see nothing here that couldn't be done in any modern engine out of the box though. They can all stream assets in and out of the level asynchronously (for "infinite extent" levels), build meshes/textures programmatically, support fully dynamic lights, etc. If you have some requirement for procedural generation that isn't supported out of the box (e.g. baking lightmaps at runtime), surely it'd be easier to just write that part yourself and inject it into the existing pipeline.
There’s nothing wrong with building your own engine but it and the intended game are going to need to be constrained.
We only hear about the successes in either camp. To get an accurate picture, we'd need to look at all the attempts.
See Astroneer for an example of this, the planets surfaces (which are deformable by the way) are procedurally generated, even if they are deployed from a kind of template specifying the parameters.
UE4 has a ProceduralMeshComponent for this sort of thing, not to mention procedural foliage component etc.
This circles back to another key reason I think a lot of people choose to build their own engine, many people don't want to spend time learning the ins and outs of somebody elses creation. When you build your own engine, you know the entire api in and out from the start and as it evolves, you know what it can and can't do, and you know where to go if it's missing a feature or has a bug.