Hacker News Reader

Comment by rakoo | Hacker News Reader

Sure, if you place yourself in an arbitrarily hard problem, it takes a lot to solve it. "How we dug a 100m pit without using machines in 2 days" is an incredible feat, but the constraints only serve those who put them.

Serving large content has been solved for decades already. It's much easier and reliable to serve from multiple sources, each at their maximum speed. Want more speed ? Add another source. Any client can be a source.

Netflix artificially restrains itself by only serving from their machines. It is a very nice engineering feat, but is completely artificial. As a user it feels weird to think of them highly when they could just have gone the easier road.

View on HN

This just isn't true though. I worked at a relatively minor video streaming company and we overloaded and took down AWS CloudFront for an entire region. They refused to work with us or increase capacity because the datacenter (one of the ones in APac) was already full. This was on top of already spreading the load across 3 regions. We only had a few million viewers.

We ended up switching to Fastly for CDN. There's something hidden here though that becomes a problem at Netflix size. We were willing to pay the cloud provider tax, and we didn't dig down into kernel level or storage optimizations because off the shelf was good enough. At Netflixes scale, that adds up to millions of extra server hours you have to pay for if you don't do the 5% optimizations outlined in the article.

You still have the same constraints: only you can serve content.

The solution I'm talking about is bittorrent. The more people watch your content, the less your servers bear load. That is using the internet to its best potential instead of reverting back to the centralized model of the big shopping mall and its individual users.

Plenty of people have tried to build a peer to peer streaming system. Amazing tried for years.

They work like garbage in practice for mainstream users.

The constraint is profit. Sure, with unlimited money you can just keep getting more and more servers. But that costs money. It would end up swamping any profit to be made.

By creating this optimized system, it makes serving that much video profitable.

No, the constraints is only you serve content. But once the content is distributed, anyone else can also distribute it.

I'm curious as to who you think would pay for the video if anyone could distribute it and watch it.

And break the profit and also probably legal constraints. Good job now you don’t have a company anymore.

How would you do it if you had much more modest scale requirements? Say a few thousand simultaneous viewers. I'm kicking around an idea for a niche content video streaming service, but I don't know much about the tech stacks for it.

A few thousand?

Just use Nginx and a backend lang of your choosing.

Not even bother using a cdn?

For low-traffic niche content that might not be a cache hit in the first place in every region?

I wouldn't bother. Unless you use storage at the CDN - which is probably very not cost effective for you.

Use bittorrent. Every viewer is also a source. The more people watch, the less your servers are loaded.

Bittorrent is built towards "offline" viewing. Try Peertube for a stack that is more built for streaming and has bittorrent sharing built-in (actually webtorrent, because the browser doesn't speak raw TCP or UDP, but the idea is the same)

Reply on news.ycombinator.com