20 karma · joined September 13, 2012
Our V2 model is trained on specific durations of audio (2s, 5s, 10s, etc) as input. So, if give the model a 7s audio clip during inference, it will generate lower quality videos than at 5s or 10s. So, instead, we buffer the audio to the nearest training bucket (10s in this case). We have tried buffering it with a zero array, white noise and just concatenating the input audio (inverted) to the end. The drawback is that the last frame (the one at 7s) has a higher likelihood to fail. We need to solve this.
And, no shade on HeyGen. It's literally what we did before. And their videos look hyper realistic, which is great for B2B content. The drawback is you are always constrained to the hand motions and environment of the on-boarding video, which is more limiting for entertainment content.
I would be curious if you are getting this with more normal images.
Also, I think it will cost around $100k to train a model at this quality level within 1-2 years. And, will only go down from there. So, the genie is out of the bag.
So far, we have purposely trained on low resolution to make sure we get the gross expressions / movements right. The final stage of training with be using higher resolution training data. Fingers crossed.
Also, Loopy is not available yet (they just published the research paper). But you can try our model today, and see if it lives up to the examples : )
The rest of our website still uses the V1 model. For the V1 model, we had to explicitly onboard actors (by fine-tuning our model for each new actor). So, the V1 actor list was just made based on what users were asking for. If enough users asked for an actor, then we would fine-tune a model for that actor.
And yes, the 7s limit on v1 is also a problem. V2 right now allows for 30s, and will soon allow for over a minute.
Once V2 is done training, we will get it fully integrated into the website. This is a pre-release.
In the AI/research community, people often try to use the same examples so that it's easier to compare performance across different models.