Hacker News Reader

Comment by onemoresoop | Hacker News Reader

Did anyone ever thought of creating an external index of youtube videos so the recommendations are open source and aren't controlled by Youtube? Basically a table with links and other metadata

View on HN

Doesn't seem like it would be easy without a firehose of metadata and I doubt Google would be amenable to providing that for a competing service.

At this point to enable that kind of competition would have to have legislative protection (e.g. a law similar to the UK one mandating banks/gas/electric companies have to provide APIs).

Selenium, click a video, store the suggested, verify what has been and hasn't been watched, move to the next, do the same etc... etc... Seems pretty easy to build a bot to do such a thing.

I suppose other search engines index YouTube all the time and they don't explicitly disallow it:

https://www.youtube.com/robots.txt

With the amount of raw footage coming from YouTube any given minute, I'm not sure there are many companies beyond Google that could do it.

The hardest part of that volume is is the footage itself. Sure archiving/serving footage may be hard, but if you just wanted to index all the titles/creators/metadata and maybe comments, I don't think it'd be that hard, in terms of volume at least.

Good clean access too it, especially if google is running any anti-scraping will be the harder part.

Most of that is in the HTML so one wouldn't even have to wait for JS execution on the page load. In fact, it's simple to download the entire page so you're not hitting up Google multiple times for each User. A few million hits over a few months and nobody will know right?

Reply on news.ycombinator.com