"The point" was to find a large project to test import performance, and it sounds like they hit that point dead center.
Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck.
I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile.
Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though.
But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost.
There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
My understanding is, that you need to reproduce the rendering bug which crashes blender to be able to fix it. And being able to reproduce it, needs to be fast. Even if you have a smaller scene which would trigger this bug, without the optimisations it would take a lot of important time in the feedback loop. Now you have a workflow which crashes the rendering in less than 2 minutes.
https://news.ycombinator.com/item?id=32191173
>At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?
It is impossible to hold a typical VFX scene in RAM to start with.
Even freelancers doing sim/FX work now have at least 128GB RAM and this is often just enough for proxy work that still gets expanded at render time.
I.e. consider the possibility that your worst estimate of how complex this data could be is off by 1-3 orders of magnitude.
And for your example: the person who signs this off is the VFX supervisor. They don't sign off anything but final frames.
Making the application wait extremely long or even crash by importing _something_ is a bug in my understanding. Why shouldn't it be fixed? Why shouldn't developers improve performance and blog about it, so other devs learn from it?
It's not about the Moana scene, that's just the test case, so OP has a valid benchmark with human comprehensible durations. The scene could be anything that is smaller and it will be imported faster now.
He does address that the crashing also needs to be fixed but that he doesn't want to be the guy doing that.
Performance improvements like this expand the horizons of what is reasonable to do.
There’s a circular logic problem where you argue that performance problems shouldn’t be fixed because nobody actually does this, but nobody does it because of performance problems. No, this effort didn’t solve every performance problem required to do a thing, but did it solve some of it.
“We didn’t do everything so we should have done nothing” is not how you build a performant product.