Hacker News Reader

Comment by trunnell | Hacker News Reader

Amazing to read about the continued re-invention of this part of Netflix's internal systems.

I worked with this team (and its predecessors) during my time at Netflix. They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds. Doing any of that in a research lab would be a challenge, but pulling it off at Netflix's scale is epic.

You might need some background on how adaptive video streaming works to fully grok this article.

But this is also just a story about a massive refactoring of a large, critical system. How many companies have you worked for that aggressively pursued refactoring/re-engineering their central systems? At most other places, I've seen risk aversion, fear, and mismanagement conspire to kill innovation. Not so at Netflix.

View on HN

Pornhub serves more diverse data across the world with a much simpler pipeline. They are the ones who should be emulated. Netflix needs a bit of housecleaning to get rid of this faction in the company.

Adult website stacks are underrated. They manage to be scrappy and yet deliver a lot.

I've seen streaming services serving half a million HD video users a day running on 7 servers, 2 of them being cheap vps.

Yeah, funny what a requirement for sustainable business models + no glut of resume-driven development will do. Almost like the good old days.

Also an open mind in hiring. Nobody will reject you because you look weird or are awkward.

As a customer I can't really say that I can tell any difference between all the streaming services, so presumably this type of optimization helps Netflix's bottom line somehow?

Netflix beats others when it comes to low bitrate high quality streams. At some remote areas with poor cellular coverage, Netflix is the only service that provides a working video stream.

So it only solves a tail problem. I would assume the percentage of subscribers with poor internet is low.

Why would you assume that?

It also helps Netflix. Lower bitrate means less server load and lower internet bills.

That interesting. I've noticed that Amazon always takes longer to load and spend more time at a poor bitrate, while Netflix just works.

Honestly, many business could learn how to beat-off the competition by emulating the adult industry.

They're often first to come to market, aren't intimidated to try new things, resist being bound by incumbents or gagged by censors and are generally wide open to grasping the full thro ... that's enough now.

I see what you did there. Like you had that one ready to pop.

Always locked and loaded me

Sex.

where can one read about pornhub's infra? does mindgeek have an engineering blog?

You are so right, I have no idea who is running other streaming services but its pretty obvious they have absolutely no idea what they're doing when they're doing bullshit like pushing blocky h.264 streams to iPhone 15 Pros with AV1 and other crap that costs a shit ton of money for a worse outcome.

Paramount looks like Youtube and uses twice as much bandwidth, that is almost impressively bad.

my 0.02 on the matter - i can commend that they got a reliable system but the deliverables are not quite comparable imho.

not all "HD" is made equal. bitrate is a major component of the perceived video quality. some experienced it first hand when youtube premium started pushing higher bitrate version for 1080p (https://news.ycombinator.com/item?id=34918698).

(disney+) hotstar is an indian streaming service which recently handled more than 50M+ concurrent viewers (https://news.ycombinator.com/item?id=38344265) for a live sports event. but if you actually watch the stream, it is of a very poor quality.

long story short: hub pushes for bitrate/quality much lower than netflix needs to in order to satisfy its paying customers, especially when they discriminate based on how much one pays.

but i wonder how much it would help with non-cpu scenarios when we need to do transcoding for upcoming codecs which might benefit from using hardware accelerators.

I'm curious about grain encoding - did you work on that at all? My friend heard they were doing a grain extraction layer that was re-added client side. Feel free to contact devin@techcrunch if you have any interesting insight.

Yes, that's part of the AV1 specs actually. See https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf

Andrey (who works for Netflix) drove the effort. Chatted with him about it.

I used to do a poor-mans version of that with low-bitrate mpeg4 (ASF or divX) movie downloads back in the day; the resolution would be low, and the MS MP4 codec tended to blur as well (plus noise killed the quality so much that there was often a denoiser in the encode pipeline), so adding any noise to the playback significantly improved the look of the movie.

Removing Grain and then re-adding it after encoding isn't specific to Netflix or even AV1 as some have suggested. I believe a recent part of H.265 HEVC, or H.266 VVC or MPEG-5 has something similar.

I‘m not impressed by the quality Netflix gives me when on my phone, all while paying for a 4K plan. Streaming a locally hosted 4K file looks just way better. Apple TV+ is approaching that level of quality, so it’s definitely doable.

That has more to do with Bitrate. Giving enough bitrate your video will be great even with older Codec such as baseline H.264. AppleTV+ uses anywhere from double to quadruple the bitrate. So it is much more about cost of business rather than technology barrier.

And it’s acceptable to bill me €16 every month and then only stream full res when I happen to be on a big screen device?

>They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds.

Both being extremely hard problem but I wouldn't say they are "holy grails" level. The latter part is currently being offered by other encoding services. While the first one, VMAF, is much better than all other metric we were previously using, still has many short comings. I was actually hoping all the AI / LLM model we could have something better than VMAF by now.

Are there other examples of LLMs discovering novel high-performing algorithms, models or metrics? I didn't think they were close to that yet.

That sounds like an epic series of projects to have gotten to work on.

Reply on news.ycombinator.com