Twitter will open-source Storm, BackType's "Hadoop of Real-Time Processing"
readwriteweb.com
readwriteweb.com
If you have any questions about Storm, feel free to ask me here or on the Google group (http://groups.google.com/group/storm-user).
Despite having a "master" node, it sounds like this actually has no single points of failure. Since all the state for the master is in ZooKeeper, I think you could just fail over to another server running the master if your first main one gets messed up. Pretty cool. (I may be totally wrong here ... All my distributed systems knowledge has come from being around people who know about distributed systems.)
Though one thing that's not especially satisfying is that if his answer for when a Bolt needs to store state is "use a database". I guess the Hadoop answer is that if your reducer fails, it just runs it again, and there's no real analog to doing that when your bolts are meant to run infinitely
It's nice to even think about opensourcing things and even better when they though :-)
Don't get me wrong - the description sounds great and I'll be one of the first users. But this is not techcrunch.
Every business should be looking at their weaknesses. If you are using someone else's service at the core of your business, that could go away at any time- you should have a contingency plan. The same for if you source something for manufacturing from another country, or have your factories in another country. What if that country becomes unstable? What will the impact on your business be?
There is nothing wrong with building your business on something that might disappear, you just need to understand that.