Hacker News Reader

Comment by bodegajed | Hacker News Reader

That's one big data. Do you have firehose access?

What do you mean? We just display data from twitter stream. You can use their stream if you want to

Which stream are you using exactly. Are you applying any filters? Afaik the complete firehose has restricted access, or did that change?

Edit: very cool visualization btw. What are you using exactly on the backend for stream processing?

I use the public stream with a filter on location (:locations => '-180,-90,180,90') Twitter doesn't seem to have limitation for that.

The backend is written in Ruby. The daemon to index tweets use the great Intridea gem https://github.com/intridea/tweetstream.

The page http://onemilliontweetmap.com/ is simple Sinatra application, and I use REDIS pub/sub to push tweets to the browser.

There's two different HTTP Streaming endpoint for Twitter's stream

https://dev.twitter.com/docs/api/1.1/get/statuses/sample

Returns a small random sample of all public statuses. The Tweets returned by the default access level are the same, so if two different clients connect to this endpoint, they will see the same Tweets

https://dev.twitter.com/docs/api/1.1/get/statuses/firehose

Returns all public statuses. Few applications require this level of access. Creative use of a combination of other resources and various access levels can satisfy nearly every application use case.

Reply on news.ycombinator.com