Hadoop: When grownups do open source | Hacker News Reader

Hadoop: When grownups do open source

theregister.co.uk

61 pointssanj42 comments

The Register: When children do journalism.

He may have some points in there, but any real content is drowned by the noise of the baseless insults, ad-hominem attacks, and poor writing. The angry rant style can be made to work, but it needs to be backed up by actual skill at writing, and it definitely doesn't substitute for content.

This article makes me sad.

And yet, we immediately recognize the style of the article is uncov. And we recognize the author's name.

It may be bad writing, but that doesn't matter. It's good enough.

No, it's not. It's sufficiently bad that I won't waste my time reading more articles by that author, and I will probably take a pass on articles by the Register, since their quality standards are low enough to let that drivel on.

(Actually, I've been avoiding The Register for a few years now, but I gave them the benefit of the doubt when this appeared on hacker news. It was a mistake.)

With writing (or anything), if you can't say something with a calm demeanor, it's probably not worth listening to.

Rants, raves, or a negative, cynical tone suggests an ill-conceived idea; it's likely an emotional reaction.

I think the idea was to be entertaining.

If you read my first post, I'm not objecting to the tone so much as the way that it's used to cover the emptiness of the article. I can appreciate a good angry rant -- I've written some myself, and I tend to enjoy Zed Shaw's rants -- but an angry rant that slams things without justification is simply childishness and poor journalism. Sorry.

Edit: Oops, I totally misread your post. Sorry about that. For some reason I thought you were justifying the tone of the article. I hear that learning to read can solve this sort of problem ;-)

I agree that a good writer can discuss something without an angry rant, but I disagree that an angry/cynical tone implies an ill-conceived idea. I'd suggest that bad writers will try to use the angry rant format to hide the poverty of their ideas, but just as often a good writer will use it well as a rhetorical device, making an otherwise dull article more amusing, and driving home the important points. But either way, the format of the writing is (or rather, should be) secondary to the content.

Edit: Clarified what I wrote a bit

I was making a general comment. Not attacking the article, or your comment in particular.

Does anyone disagree that an angry rant, as a rule of thumb, probably means an ill-conceived idea?

EDIT: If you misread my comment, that's my fault. I should have written better.

I disagree. It is often the result of a frustrating experience.

Nah, I was distracted when I read it. It's clear, I'm just stupid.

Often valuable information is conveyed in a not-so calm demeanor. The delivery may not be the best, however.

I think you misunderstand me.

I don't think the article is great. But it -does- accomplish exactly what the author wants. People talk about it. It reaches the top of HN. And despite of our distaste of The Register... we still read it.

So when I say it's "Good enough" it just means it gets the job done. The purpose of the article is clearly attracting attention. Being ignored worse than being disliked, after all. And it does attract a lot of (negative) attention. The poor writing even contributes to the goal.

Read it for entertainment purposes only and it becomes... better.

Along with the data processing framework, Doug Cutting also included a fault tolerant, replicated, distributed file system with Hadoop just because fuck you. -- I laughed!

Also, it makes fun of Ruby so another point for it.

It's unlikely this article is accurate or really serves any purpose whatsoever other than immature linkbaity sort of content, but that's fine with me because I never put it in the category of "real journalism."

The article was definitely entertaining. I definitely laughed at

"Starling is the Ruby-based messaging system that runs Twitter's backend. Yes, Twitter, the nonprofit web service known widely for its downtime, dropped its disaster-producing shitpile on the world."

I agree. It's not like Hadoop and a related project Lucene (by the same lead programmer) don't have their own share of problems at some point in time.

(For the record I like Hadoop and Lucene.) I still remember the concurrency problems Lucene had a few years back which would result in index corruption (a lot of Java programmers experienced this). This was the reason that while I was excited about Hadoop when it 1st came out I was also hesistant. It was by the same guy who didn't fix concurrency problems for his previous major project - would history repeat itself? Well Hadoop first came out it had a similar problem which caused it to fail a task in some scenarios.

All of this may have been fixed by now, but I just want to show that nothing is perfect (regardless of language) and unless you are; you really shouldn't be a jerk about other people's hard work

This guy is so annoying. You might think that the non-success of his own startup (http://www.techcrunch.com/2008/07/19/pressflip-is-a-belly-fl...) would make him slightly more humble, but apparently not.

Maybe he decided to focus on his most sellable skill.

Yes, I apparently those that can, do, and those that can not, talk trash.

And people complain about Zed Shaw's expletive-filled writing style? At least Zed has a sense of humor and blasts just about everything in sight. As for this article, let's see. Hero worship? Check. Ad hominem attacks? Check. Uninformed comments ("RubyForge? What... is that?")? Check.

Yeah, this is disappointing. Mostly because Hadoop is, in fact, a pretty nifty piece of software. But it's a cluster thing, and individual hackers don't do clusters for some pretty fundamental economic reasons. So to Dziuba, that means that the Hadoop people at Yahoo and elsewhere are "grownups". Sigh.

I thought that was an odd distinction. For some reason, he equates open-source with the Web 2.0, Ruby crowd. The first things that come to my mind are Linux and gcc, which are two of the most important software projects on the planet.

No, he doesn't. He's clearly drawn a distinction between Web 2.0 open source and the "rest" of open source. His entire point is to belittle and laugh at Web 2.0 open source attempts.

And it seems that unlike Zed Shaw who wrote the webserver behind almost every Ruby site out there these guys' technology contributions are...a digg-based search engine.

Does anyone on HN have any first-hand experience configuring and using Hadoop? If so, how was the process - was it straightfoward or complex, did it require lots of peeking into the source, any major pitfalls, etc - and, are you satisfied with how it's currently working and with your ability to add new boxes easily?

Hi,

I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs.

As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster.

If you have these specific needs: a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?) b) ability to add storage for new data without powering machines down c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop).

Now, if you still think you need hadoop (in most of the cases, answer is "you don't"):

To get hadoop going minimally to be useful right away is actually almost trivial.

However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome.

To answer your questions directly: - Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward. - Peeking into the source is not required to get it going - Pitfalls: * Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory) * Find the optimal size of input data for your task (if you are using compressed files for input).

(Yes, I will write these learnings up eventually).

Great reply. My current thinking for the project I'm working on is that I don't need Hadoop...yet. That said, if it's trivial to implement on a single machine it might be wise to just begin the project in Hadoop so that I don't have to make the switch later. Always balance between 'premature optimization is the root of all evil' and 'choosing the right tools for the right job'.

I'll give it a shot on my desktop to start - thanks for the tips!

Posted some thoughts on what problems are appropriate for Hadoop here: http://weblog.strlen.net/?p=8

(Another post on parameter settings and machine configurations for Hadoop cluster in the making, once that's done I'll post to news.y.c)

Wow. That might be the most unprofessional thing I've ever read.

I, on the other hand, have read other articles from The Register. This is par for the course.

Heh - I'm working on a MapReduce blog post as we speak (see http://tinyurl.com/6oylnr). So I suppose I'm prone to his attack.

I actually think he makes one good point, that virtually no one actually needs to use MapReduce, let alone implement it, let alone write a blog tutorial on how to implement it. That said, virtually no one needs to write a compiler either, and yet understanding compilers (or even writing one) is not a waste.

Also, his comparison of a message queue (Starling) to MapReduce almost makes you wonder if he really understands MapReduce. Starling has nothing to do with MapReduce, and MapReduce is completely unrelated to Twitter's scaling problems.

And, Starling is not the reason for Twitter's scaling problems. It's actually a very very simple and very useful and it works as advertised.

The question I had is whether Dziuba built something interesting. Zed has.

He built a search engine called Pressflip which has a neat idea(say which searches are good or bad) but fails on the execution.

Parts of the article made me laugh, especially the Twitter as drum major quip (me being an old band/drumcorps member). Not knowing much about hadoop or map reduce, I gather he doesn't really have a clue based on his dismissal of Twitter as a failure. Last time I checked, they're going strong.

I hope someday I manage to build something that people laugh at as much as they're laughing at Twitter.

This is a troll of epic proportions.

Well, not really epic. Mini-series proportion is probably more apt.

I gathered that much just from the title. If you trained a Bayesian filter to recognize trolling, "grownups" would probably be one of the strongest troll tokens. adequacy.org even subtitled itself "news for grownups".

I'm pretty much done with that e-rag.

The only thing I got from this article is reaffirmation that Ted Dziuba is a huge Hadoop fanboy and still hates Twitter.

Amazing article, only because it called out so many types of people simultaneously.

It's like Zed Shaw without the factual accuracy.

"...but rather as an act of mercy for his keyboard"

-Classic.

heh... "Along with the data processing framework, Doug Cutting also included a fault tolerant, replicated, distributed file system with Hadoop just because fuck you"

Reply on news.ycombinator.com