Comment by romaniv | Hacker News Reader

Comment by romaniv | Hacker News Reader

>Why do we need Style Transfer for Music?

We don't. Most of the applications of ANNs to music that I've seen so far are solving problems that either don't exist or have been very successfully solved by much simpler methods with much better results.

If you want to apply AI to something useful in music, how about:

- Machine learning for reverb generation. I.e. being able to "record" reverb from real space and apply it to sound.

- AI/ML for high-quality sample transposition.

- AI for pure sound synthesis. Imagine something that does PCA a set of sounds and then allows you to generate new sounds by changing components with a bunch of knobs.

- AI for real-time sample morphing. For example, being able to synthesize a choir that sings the words you need. Or, how about transforming your own voice into a quality choir vocals? Hey, style transfer!

- AI for high-quality music transcription (sound to notes or MIDI). This is being done to some extent, but I don't think it's very good yet.

And so on.

I can assure you, every singe one of these would be something musicians would be willing to pay good money for.

- De-reverberation is being intensely (in some niches at least) studied with these tools, and adding the reverb back (if you can learn and remove it from mixed records) means you have probably modeled it well enough to do whatever you want to. But de-reverberation is very hard especially with single source (1 mic) recordings, maybe looking at better reverb generation is easier with the right data. It would be nice to do something that didn't require the physical setup of impulse response recording.

- I haven't heard the term transposition for samples - is this the name for moving samples in pitch and compensating in time so that it sounds undistorted?

- On pure sound synthesis, see NSynth [0]. It's a beginning step toward this with abstract sounds.

- Sample morphing and changing the words with voices is pretty tough (overlaps neural TTS heavily, which is pretty recently starting to work well), but if you can synthesize a single voice decently the rest seems doable with manual intervention. Check Neural Parametric Singing Synthesizer [1], or try the online demo [2]

- Every few months there is a new convnet which improves the baselines for transcription, hopefully it is only a matter of time. The current best is quite a bit better than only a few years ago.

There is still a huge gap between "what is possible in research" and "what is easily usable in a practical editing/creation workflow" on a typical DAW, but hopefully that gap will diminish over time.

There are also a lot of tools from statistical signal processing that were doing these tasks to some degree, digging them back up and "neuralizing" the probability parts with neural nets is a promising way to get quality improvements, and would mirror what has been successful in neural TTS so far.

[0] https://experiments.withgoogle.com/ai/sound-maker

[1] http://www.dtic.upf.edu/~mblaauw/NPSS/

[2] https://www.voiceful.io/demos.html

>- I haven't heard the term transposition for samples - is this the name for moving samples in pitch and compensating in time so that it sounds undistorted?

Yes, that's exactly it. There are traditional algorithms that do it, but most are variations on time stretching and granular synthesis. They aren't as good as they could be. A good algorithm would be able to take in a handful of examples and generate a model that seamlessly interpolates across different pitches and velocities, accounting for differences in timbre, attack, decay, noises and so on.

Thank you for the links.

Recording reverb from real space is as simple as recording an impulse (firing a gun or anything that makes a short high intensity noise) in a room and then convolving that with the audio you want to add reverb to.

The only limits to this is how close to a perfect impulse you can generate.

You really might not even need to make a great impulse... you can probably get away with recording response to a series of "known signals" (e.g. chirp signals played back over a known loudspeaker), and then determining the impulse response from that, and applying that (via convolution) to your actual signal of interest.

Reverb of a static room at a point is as simple as that. Rooms with moving obstructions, with moving noise generators, at moving points, that's harder.

Well, yes, that's how convolution reverbs work. Except I've never seen anything that would allow users to record a reverb with conventional hardware and without any elaborate setups.

Also, there is more to reverb than just impulse response. A more sophisticated reverb simulator would map out the space and allow you to choose where the "listener" and the source are located in that environment.

The nearby comment about moving obstructions is on-point as well.

There is ample space for application of machine learning here.

Reply on news.ycombinator.com