Comment by willholloway | Hacker News Reader

Comment by willholloway | Hacker News Reader

Parent Full thread

This is an excellent post. I used almost exactly the same system to build my app streamjoy.tv . Everyday streamJoy scours Netflix, iTunes and Amazon to find the availability of movies for streaming, rental and digital purchase.

I built it as a portfolio piece and haven't finished it because I've been doing consulting jobs, but if you want to watch a particular movie online without going the pirate route, it's the start of a legal alternative to the old sites like sidereel.

The tech behind it is the same as this article. Flask and jinja render a static html page for each of about 92,000 movies.

I use a flask app for the search functionality that accesses an elastic search database.

I used mongodb because it was incredibly easy to create a local cache of the JSON data I was getting from the apis I was accessing.

It all took a lot longer than I ever would have though to even get it to this point. There were a lot of little annoying issues with the various APIs I had to access, and the annoyance of parsing XML amongst other schelps I had to deal with.

I have only ever mentioned it on hacker news one other time and the last time my elastic search server crashed from the traffic. It is all running on a $5 a month digital ocean vps.

The flask/jinja static page creation is rock solid and would never fail if I pushed it to s3, right now my elastic search server is the bottle neck. I haven't taken the time to throw hardware at it or set up clustering.

All in all it's a pretty cool service in my opinion, I built it for myself because I love movies and spend a lot of time watching them online and made a decision to never pirate a content creators work again. Also the experience of netflix, amazon and itunes is orders of magnitude better than the old megavideo/bittorrent trouble of finding the real deal and not being inflicted with spammy ads with voiceover.

I really like the flask/jinja/bootstrap/javascript/mongodb/elastic search stack. I've learned a lot of tips and tricks by building streamJoy and if people want I would be happy to share them with the community.

I know this sounds like self promotion of my app but I haven't even taken the time to implement affiliate tracking for any service besides amazon. Consulting is serious and real money right now and that takes priority over this little side project I did.

Sounds like a cool app, I'd love to hear any tips or tricks you've learned along the way

Thanks! I'm going to write up a post with more details about the tech behind it. Stay tuned.

Looking forward to it!

Out of curiosity, what do you use to get the movie images for the background?

The images are promotional images that have been uploaded to the movie database, an open source alternative to imdb.

streamjoy.tv looks nice. Too bad you're not planning to finish it. Another alternative that I've been using recently is canistream.it but that get's wonky some times.

Just curious. Could you explain this part? I used mongodb because it was incredibly easy to create a local cache of the JSON data I was getting from the apis I was accessing.

If you are using the data from the various APIs to render static pages, why do you need the local cache of the JSON data? And if you've got a local cache of all the JSON data, why render static pages rather than serve dynamic pages that reference the DB of cached data?

Sure, I would be happy to explain.

> If you are using the data from the various APIs to render static pages, why do you need the local cache of the JSON data?

Because the data that I am displaying for each movie is not available from any single source. streamJoy is the abstraction layer tying together the disparate data sources.

Mongo was the right choice because instead of having to map out postgres schema, I could just store the entire JSON response as a dict.

What I have learned from all of this is that API's are not always reliable. The information changes, you don't always know what you are going to get back.

Mongo just made this early process where I didn't know what I was going to get much more fault tolerant. And, no schema mapping.

> And if you've got a local cache of all the JSON data, why render static pages rather than serve dynamic pages that reference the DB of cached data?

Rendering static pages allow you to use S3 or nginx to serve your html. For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files.

I'm using nginx right now only because I haven't taken the time to use S3, but S3 is the smarter choice here.

My goal with this was to have the smallest possible dynamic server footprint as possible.

The other reason for caching json data is I run analysis on the db items, even though I haven't published any of those features yet.

By having my own copy of the data, I can run a process on 92000 items in 6 minutes instead of taking a day due to API rate limits.

> For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files.

Have you tried putting CloudFlare in front of it? Buttered jelly-roll manna.

You can accomplish the same thing as flat files by using something like Amazon CloudFront (http://aws.amazon.com/cloudfront/) -- CloudFront caches your dynamically generated pages and serves the generated page from cache rather than hitting your server each time.

See http://aws.amazon.com/cloudfront/dynamic-content/

Reply on news.ycombinator.com