Hacker News Reader

Comment by imchillyb | Hacker News Reader

Not OP, but...

Look at the list of what Apple Sing includes:

Adjustable vocals: Users now have control over a song’s vocal levels. They can sing with the original artist vocals, take the lead, or mix it up on millions of songs in the Apple Music catalog.

Real-time lyrics: Users can sing along to their favorite songs with animated lyrics that dance to the rhythm of the vocals.

Background vocals: Vocal lines sung simultaneously can animate independently from the main vocals to make it easier for users to follow.

Duet view: Multiple vocalists show on opposite sides of the screen to make duets or multi-singer tracks easy to sing along to.

------

The part of the article where they state these things explicitly...

View on HN

> Adjustable vocals: Users now have control over a song’s vocal levels. They can sing with the original artist vocals, take the lead, or mix it up on millions of songs in the Apple Music catalog.

I think this only requires pre-making two audio files per track, and simultaneously streaming these.

Real-time lyrics, Background vocals and Duet view are all nice features too, but the hardest part processing-wise is analysing how loud you sing into the microphone. It's just karaoke with a good UI.

They say it supports millions of songs so I doubt that’s how they are doing this.

They are likely using a sophisticated ML version of what old karaoke machines did, and removing the vocals in real-time.

Even if they're using ML, I don't imagine they'd do it on-device. They don't even do voice recognition on-device by default.

> [Apple says it is] relying on an on-device machine learning algorithm that processes the music in real-time. The tech builds on Apple’s noise-cancellation expertise and other developments it’s made for FaceTime, the company said.

Source: https://techcrunch.com/2022/12/06/apple-music-is-getting-a-n...

Ah, I stand corrected! I wonder why they took that approach...

Not sure - although I would imagine that it would effectively double the storage and bandwidth/data requriement for Apple Music in general if they had to send two files with equal bitrate.

I stand corrected as well!

Wonder why they take this approach though, as it is clearly over-engineering (if I correctly understand that the goal is just to make vocals volume adjustable).

Maybe it's licensing? I can imagine copyright holders being squeamish about Apple processing, permanently storing, and serving heavily altered versions of their music. The difference is silly and pedantic, but by processing it in real-time during playback, one might argue it's just a filter effect like EQ.

> Wonder why they take this approach though, as it is clearly over-engineering (if I correctly understand that the goal is just to make vocals volume adjustable).

Depends what the other non-functional requirements were. i.e. if the NFRs were as follows:

* Cannot increase bandwidth / mobile data usage.

* Cannot impact music quality / bitrate.

* Has to work offline.

* Cannot increase on-device storage.

* Has to be responsive.

Then two audio streams might not work.

Another advantage of doing it on-device is that it doesn't actually change any of the backend architecture too. It might be a lot of change to a lot of systems for a feature which only adds a small amount of functionality - i.e. architecting your entire backend and streaming around seperating audio tracks might not be the right focus.

> I think this only requires pre-making two audio files per track, and simultaneously streaming these.

That’s the understatement of the century. “this only requires […] simultaneously streaming these”

Well, it is indeed tricky but not requires-dedicated-chip tricky :-)

Don't movies already have stereo sound? Is going from two audio channels to four really that difficult?

They don't state anything explicitly referring to real-time processing of the songs.

As a matter of fact, calling out "millions of songs in the Apple Music catalog" actually makes it seem like the adjustable vocals will only be available on certain songs that they've added support for.

I’d guess it’s more probable that lots of songs simply have no lyrics at all, so the claim „all songs in the Apple Music catalog“ would be factually wrong.

It's hard for me to imagine they'd do something special to support "millions of songs" while excluding others, since the entire catalog is ~100m songs.

My guess is that it's entirely dynamic. It's hard to imagine the complexity of doing batch processing to render each song in the library, and maintain that as new songs are uploaded, and update the renders for software improvements. Better to just do it realtime.

And since classical, instrumentals, esoteric ambient stuff, death metal, etc, will probably not be supported by the algorithms, I think the "millions" refers to those that can be processed in realtime.

They do something special: get the necessary sign offs from legal. Apple‘s license for some content may bar it from being used that way (that’s another reason for doing it on device: could be different situation as far legal & licensing are concerned)

Sure, and a good point. I should have said they wouldn't do anything special on a song-by-song basis for millions of songs. It's not like someone's pushing the button for each song, or building a list of songs. Those that meet the criteria will be included, whether that is 10 or 10 million of them.

Why would you need a dedicated chip for any of that?

Seems like it is powered by:

> [...] an on-device machine learning algorithm that can process music in real time

Their latest apple TV includes the A15 which includes a 'neural engine' for ML, and this is also included in their latest iPhone / iPad, so that might be part of it.

The divide in expectations is funny. Non-Apple-user: "ML stuff? Must be 'in the cloud'." Apple user: "ML stuff? Must use a special chip in the device."

And then there's me, asking why do we even need ML for that. :D

Reply on news.ycombinator.com