Netflix: Lessons in Experimentation

If you want something selected and curated by actual humans, I recommend checking out Mubi. Lots of odd ball, but phenomenal movies on there. They started out having thirty movies on there and replacing the oldest every day with a new movie. It's often times a few in a row by a specific director or around a theme. They now also a have decent library in addition.

Almost every movie I find truly memorable in they just few years came to me via Mubi.

Recommendations aren't really chosen by "experiments". And certainly, they aren't A/B testing budgeting and development of shows.

Recommendation engines can be improved with experiments. And they - along with budgeting and show development (at least at Netflix) - are "driven by big data".

Further, it doesn't matter if you - personally - like a feature. If that feature is moving the numbers and passes the A/B test, it will make itself into the app.

They run thousands of experiments. Of course there are going to be several that you - personally - do not like.

> why are the recommendations so bad

Without knowing anything about how it works internally I'd guess that the recommendations are not simply about what's best for the user but also what's best for Netflix (pushing Netflix originals).

They used to have good recommendations for their DVD service with star ratings. The rental catalog was huge and varied then so a lot of interest areas were well covered and easy to extrapolate from your ratings and your common cohort. I suspect the problem they have now is most of the top tier content is gone and it's 90% filler that isn't compelling to watch.

I can only assume that the majority of people like a) rewatching things they’ve already seen, b) like watching things they said they don’t like, and c) watch things from categories they don’t like.

Because that’s what Netflix has been doing to me pretty much since they became a streaming service.

> If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List?

They've gotten very good at grabbing attention with cover art, but then you click on it and quickly realise its crap.

In other words, they've put appealing covers on all the boxes, so now you pick ones that are crap only to abandon them 20 minutes later.

I wonder if abandonment rate is a metric too.

I think the recommendations are not good on Netflix because the content is mostly not good. Lots of Netflix Original stuff that lacks staying power and seems designed to pander and not to actually be good. Fortunately, as time marches on, Turner Classic Movies is starting to show films from the 70s and 80s (I guess something can only be a "classic" if it's at least 40 years old), so I imagine that TCM will get a better and better selection in the coming years. If you have a cable subscription, you can stream their rotating selection of films online.

I also find that Hulu generally has better options for TV series than Netflix. You have to deal with ads unless you pay extra for the premium subscription, but it's not like they aren't harvesting your data anyway, and it's easy to just mute the ads (I always muted them on cable, too) and go do something else. It's also still a lot fewer ads than on cable TV.

It's been pretty apparent for me that their who focus changed a couple of years ago from "here's some stuff you might enjoy watching" to "here's the stuff we want you to watch" or much like Google ads, "here's something you've already watched, watch it again"

I'm almost at the point of cancelling my Netflix subscription simply because it's so much effort to find something of interest now.

There already exists a simple icon for "I don't want to watch this": the thumbs down button. Clicking it is supposed to remove the show from your recommendations (after a day or so, to clear the caches).

For the love of God, can we please get an "Only show me things in my native language" option?

Yes, someone with some basic recommendation engine knowledge please apply! A basic rip-off website streaming site has a better recommendation engine AND ux then this company. Was given a subscruption as a present and it dumped it in the bin a few weeks later after. It seems like the algorithm is a basic highlight whatever scam show is pumped by the bingers at the time. Not even the show i last watched (consitantly everyday) is easy to navigate back to if u close the app. I swapped back to the streaming sites and found more NetFlix content immedataly.

Thx for observables though!

Do create a button called "wipe my recommendations and start anew"

You might be interested in Trustworthy Online Experiments by Kohavi et al then. I enjoyed the historical progression!

We've recently done a large blog series on experimentation. https://netflixtechblog.com/netflix-a-culture-of-learning-39...

Kinda covers everything and links out to posts that cover the software engineering in more detail and papers that cover the math in more detail.

The reason they're there is that they are known primarily for the high salaries they pay.