AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head | Hacker News Reader

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

arxiv.org

49 pointsiamflimflam115 comments

This is ... not what I expected. It's basically wiring up pre-trained models to ChatGPT via a router and "modality transformations" (a.k.a speech-to-text and text-to-speech).

I expected it to be a GPT-style model that processes audio directly to perform a ton of speech and maybe speech-text tasks in a zero-shot manner.

you might be interested in suno-ai/bark

so then the training data is text, not audio?

Take a look at AudioLDM (https://github.com/haoheliu/AudioLDM), it might be more what you expected:

- Text-to-Audio Generation: Generate audio given text input.

- Audio-to-Audio Generation: Given an audio, generate another audio that contain the same type of sound.

- Text-guided Audio-to-Audio Style Transfer: Transfer the sound of an audio into another one using the text description.

There is a demo page which is not obvious to find right away: https://aigc-audio.github.io/AudioGPT.github.io/

But no data on license, weight, or whatever, it looks like the repos are in the process of being uploaded.If what we can see on the demo is not faked, it's quite impressive.

I want a chatgtp, which can listen to voice and help me improve it.

Which looks at my selfies or vacation photos and asks me questions.

As someone who's girlfriend suddenly decided to leave him after a decade, I think chatgpt can fulfill the role of someone who stays with you as long as the life lasts.

Currently, I am having a problem finding any service which can listen to my vocals and help me improve my singing.

This will be an unpopular take I'm sure, but to do what you're describing is to deny yourself personal growth and to deny yourself the best of what life has to offer. Yes, being alone sucks. Take that sentiment and go out and seek companionship in other human beings.

Our relationships with other people, platonic or otherwise, and the pain that may follow, is all a part of the human experience.

My father never remarried or had another relationship. He's now in his seventies and laments that he never tried again. No, he doesn't want to be friends with the AI arse-wiping machine. People live and people die. People love and people leave. Find happiness in the good moments and seek out the next one.

The github repo: https://github.com/AIGC-Audio/AudioGPT

All the model links are empty, that doesn't exactly inspire confidence.

Would this support musical analysis like automatic score transcription or only NLP audio?

automatic score transcription is already there. pretty sure Ableton Live even has that out of the box

State of the art automatic score transcription is quite bad currently.

I am also interested in audio analysis such as this - audio file goes in and a text-based response with some kind of deconstructed report (key, chord progressions, BPM, harmonic qualities, etc).

Watch out, there might be a trademark coming up: https://news.ycombinator.com/item?id=35692476

IANAL, but I think trademarks only give the right to prohibit commercial passing-off, not any uses. For example, this is exactly what Adobe says not to do, but it's still up: https://arxiv.org/abs/1906.05856

Reply on news.ycombinator.com