Show HN: Caramel: a speech-to-speech conversation model trained only on audio

I trained a speech-to-speech model last night, inspired by Dessa. I'm calling the model caramel-1. The model learned from recordings of people talking and nothing else, and has never seen a single word of text. The code, the training recipe and the weights are open source.

caramel-1 sounds incredibly human. Most of what caramel-1 says doesn't make sense yet, but the sentences make a little more sense every time I add more data.

Caramel is fast. On a GPU, caramel-1 makes 1 second of audio in about 0.09 seconds, which is fast enough to talk to a real person live. It even works on my MacBook Pro (M3 Pro): running two copies at once, it makes 1 second of conversation in about 0.25 seconds.