I don't think it rigs the models - I think that video is comprised of models generated by Stable Zero123 that were then rigged/animated/postprocessed in Blender.
75 karma · joined September 28, 2022
You can get something up and running - but it feels akin to staring at a 1080p screen with a weird delay. I felt way more productive just looking at my tiny laptop screen.
Your mileage may vary, obviously. I really liked the idea of working in a distraction free virtual environment, but I feel like the tech isn’t there yet unless you’re an enthusiast whose willing to put up with all of the strangeness.
There’s plenty of papers that have tried text -> 3D model generation using photogrammetric-esque methods similar to what the parent comment suggested - the two minute papers video on one example is here.
Outside of cherry-picked examples this style of model tends to suffer from what people are calling the “Janus Problem” - the easiest way for it to satisfy the loss is to simply make the object look like the input prompt from as many angles as possible. So if you enter “a rubber duck”, it tends to generate yellow blobs with multiple head-like appendages sprouting off from it.
Google’s paper that tried this approach using Imagen as the text->image generator had great results, but they might be cherry-picked. Someone replicated it with stable diffusion as the text->image backend - still major Janus problems.
Are you saying they should have more bombast in their product announcements?
See Imagen for example - https://imagen.research.google/
If Google is to be believed, this outperforms Dall-E - and I’ve heard from people that use it that in general, it does perform better than Dall-E.
At this point we have a public robotaxi service in one suburban area and two robotaxi services operating under restricted domains in SF that are open to the public behind a waitlist. The cars have problems with rain, fog, or snow, and expanding to a new city still takes a ton of time. Having joined the industry after the hype of 5 years ago, the reality now is nowhere near the vision companies we’re selling back then.
My point wasn’t that these systems will never work in some capacity - my point is that they’re taking a lot longer to roll out than people predicted, and that the internals of these systems aren’t end-to-end ML models - it’s ML-based perception feeding in to a lot of traditional robotics code, with ML models handling certain prediction tasks in certain spots.
What’s likelier is that ML-based tooling becomes a key part of the animation workflow. Used to generate assets, animations, backdrops, characters, etc, which are then combined by an animator/editor. The examples he cites in the article all fit this mold - the anime character generator he cites in the article uses separate models to generate the character then rig it from facial data.
After working in the self-driving car industry, I’m really skeptical of any claim that rapid advances in one modality or task mean we’re “just 3 years away” from all related tasks being done via ML models. Alexnet et al completely revolutionized perception in self driving cars - between 2014-2017 it was really common to hear predictions that we’d have end to end models driving our cars perfectly in “less than 5 years”. That reality never arrived because ML just wasn’t capable of handling more complex tasks the way it could with object detection. Lots of articles similar to this one talking about what we were going to do with all of the out of work truckers and Uber drivers.
And yeah obviously generative art is a different beast than autonomous driving. And I’ve seen examples from WIP text2video models. But I just want to caution that tons of progress in image and text doesn’t necessarily mean we’re just a year or two away from all related tasks being conquered.
Blown away by how quickly this stuff is advancing, even as someone who's relatively cynical about AI art.