Hacker News Reader

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

We are live again folks! Sorry about that. We ran out of storage space.

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

The system just crashed. Sorry! Working on getting things live again as fast as we can!

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

Nice! Thanks for sharing. I hadn't seen that paper before. Looks like they take in a real-world video and then re-generate the mouth to get to lip synch. In our solution, we take in an image and then generate the entire video.

I am sure they will have open source solutions for fully-generated real-time video within the next year. We also plan to provide an API for our solution at some point.

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

For the input, we pass the model: 1) embedded audio and 2) a single image (encoded with a causal VAE). The model outputs the final RGB video directly.

The key technical unlock was getting the model to generate a video faster than real-time. This allows us to stream video directly to the user. We do this by recursively generating the video, always using the last few frames of the previous output to condition the next output. We have some tricks to make sure the video stays relatively stable even with recursion.

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

Yes. We use Modal (https://modal.com/), and are big fans of them. They are very ergonomic for development, and allow us to request GPU instances on demand. Currently, we are running our real-time model on A100s.

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

We just removed email signup. You can try it out now without logging in. It was easier than expected to do technically, so we just shipped a quick update.

sid-the-kidon Show HN: Lemon Slice Live – Have a video call with a transformer model

That's fair. We just removed the sign-in for HN. Should be live shortly.

Each person gets a dedicated GPU, so we were worried about costs before. But, let' s just go for it.

sid-the-kidon Show HN: Sonauto API – Generative music for developers

I am with you. I want this too. Maybe somebody can make it wit their API?

sid-the-kidon Show HN: Sonauto API – Generative music for developers

Okay. I know these guys IRL. BUT, I genuinely think they have the best music model out there. Hands down. The songs are just more unique, and have a wider range of musical variation. With Suno/Udio, the songs just sounds the same after a while (just with different lyrics).

That could just be me though. I am curious what users of Udio/Suno think?