jhammerb

Hey,

We're on iteration 2 for this course, and it's still in somewhat rough shape. If you plan to devote significant time to the lectures, I'd recommend waiting until next spring, when we'll be teaching iteration 3 online at http://ds-class.org.

Later, Jeff

Avro defines both a row-major and a column-major layout for files: https://issues.apache.org/jira/browse/AVRO-806.

For more details on Oozie, see http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-oo....

There's been some work to add indexing to Hive. See http://www.slideshare.net/NikhilDeshpande/indexed-hive and https://issues.apache.org/jira/browse/HIVE-1803, for example.

Independence from Red Hat, since Red Hat sees VMware as their biggest competitor: http://www.pcworld.com/businesscenter/article/199870/red_hat...

Facebook gave options to employees for referrals early in their history. At some point we'll switch to cash at Cloudera as well.

from my perspective, a single user interface paradigm is unlikely to cover the variety of use cases that the full suite of hadoop platform services (flume, sqoop, hdfs, mapreduce, hive, pig, oozie, hbase) provides.

at cloudera, we've built and released into open source a user interface construction kit and an environment to house these various "applications". that way we can incorporate any metaphor you'd like: file browser, spreadsheet, yahoo pipes, etc.

see http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3b2-hue and http://www.cloudera.com/blog/2010/07/developing-applications... for details.

Not true at all. They do have other lines of business, but their databases business continues to grow. See http://seekingalpha.com/article/185179-sybase-inc-q4-2009-ea..., e.g.

After participating in a few of the XLDB meetups, at which the requirements for and design of SciDB were debated and discussed, I was quite excited about this project. I was kicked off of the mailing list unceremoniously once they decided they wanted to make it a business, and I haven't been allowed back on since. They're certainly abusing the term "open source" for marketing purposes here.

The reason they don't want people looking behind the curtain is that the rhetoric far exceeds the reality.

Hive has ODBC and JDBC drivers. What SQL functionality are you missing?

Google also kept the RPC side of Protocol Buffers, which is called Stubby, as an internal project.

Everest was replaced by Hadoop + Pig, Yahoo's take on a Hive-like system. Both Hive and Pig allow you to use SQL.

I thought Microsoft did a good job of adding a procedural-like feel to SQL with SCOPE's virtual views; see http://icme2007.org/~jrzhou/pub/Scope.pdf for more.

PigLatin certainly is a nice approach to the problem as well, of course.

Just to be clear, I was never CTO of Facebook or anything close to it (thanks for that, though!). Dustin Moskovitz and Adam D'Angelo were the only two folks to hold that title that I know about.

And yeah, I don't want to get in trouble, but ask the right people about "Team America"...

Facebook certainly has the expertise to build their own servers: http://www.facebook.com/techtalks#/video/video.php?v=2085616... and they are hiring Hardware Engineers: http://www.linkedin.com/jobs?viewJob=&jobId=822679

Well, Chris Mason is focused on btrfs, not so much ext4. See, e.g., http://www.insight24.com/webcasts/content-100695_1

Excel has incorporated software from Frontline Systems, rebranded as "Solver", for many years. Most heavy Excel users use it every day. I couldn't dig through the marketing speak entirely, but this product appears to be a port of Solver to C# and not open source. Of limited interest to HN readers, I'd expect.

Okay, copy and paste from previous threads:

We're hiring here at Cloudera.

We're especially interested in web developers who have built and deployed large, extensible applications into production environments. An interest in data visualization and analysis doesn't hurt. We also have some deep distributed storage system hacking problems.

We have a strong preference for open source experience: our team (see http://cloudera.com/about) includes core contributors from the Berkeley DB, Ganglia, Lucene/Nutch, Hadoop, and MooTools projects.

We expect you to communicate ideas clearly, exhibit preternatural intellectual curiosity across a variety of domains, write quality code, and have a consistent focus on improving yourself and the team around you.

If you're interested, drop your CV and a cover letter to jobs@cloudera.com.

The early release program has already started and the product is in use.

In addition to the points below about Hadoop's implementation of MapReduce, Hadoop consists of many subprojects which perform complex tasks: HDFS, a reliable, petabyte-scale distributed file system; HBase, a clone of Google's BigTable; Pig and Hive, which provide higher-level syntax, columnar storage in HDFS, and a persistent metadata repository; and Zookeeper, a coordination service for distributed systems.

I'm not impressed by how REvolution's new management is approaching marketing.

If you have bucketed tables, you may find the TABLESAMPLE operator useful to iteratively construct complex queries and get query results back sooner: http://wiki.apache.org/hadoop/Hive/LanguageManual/Sampling. Agreed that Hive, and databases in general, could be better in this regard.

Digg will be hosting an Avro hackathon on Thursday, November 19. Sign up if you'd like to hack! http://avrohackathon.eventbrite.com/

Hey,

As a founder of Cloudera and a participant in the Apache Hadoop community, I'm dismayed whenever the press writes something about Cloudera "creating" Hadoop. No one here believes that, and we try to acknowledge the great work done by the community, especially at Yahoo!, Facebook, Amazon, Powerset, et al., whenever we can.

To the best of my knowledge, we're the only company to provide a commercially supported distribution of Hadoop. I'm pretty sure that's what Ping meant when he says "behind"--supporting, not necessarily solely creating.

Anyways, it's an open source project, and the copyright belongs to the Apache Software Foundation. In the case of Linux, no one organization contributes more than 13% of the code (e.g. http://lwn.net/Articles/222773/). The same distribution of contributions will likely be the case for Hadoop as it matures, and it's reasonably close to that state now: there are subprojects to which neither Yahoo! nor Cloudera is the primary contributor, and it is most certainly not the case that 80% of all Apache Hadoop code comes from any single organization.

Either way, Hadoop is the most interesting open source project with which I've been involved, and I hope the increasing attention doesn't distract the community from its primary goal of producing kick-ass code.

Later, Jeff

Link to paper on which article is based: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnum.... Sweet paywall.

See http://bitbucket.org/_ben_/cassdict/overview/ for an implementation of orthogonal persistence for Python's dictionary data structure using Cassandra.

most of my issues with r can be traced back to the nonintuitive (for me) data structures. i found http://www.amazon.com/Data-Manipulation-R-Use/dp/0387747303 to be the most concise guide for how to manipulate common data structures.

Follow this feature request on Get Satisfaction at http://getsatisfaction.com/getsatisfaction/topics/is_there_a....

The news article is about the joint NSF-Google-IBM Cluster Exploratory program (http://en.wikipedia.org/wiki/Cluster_Exploratory), which grew out of a Google project to bring MapReduce into undergraduate education (see, e.g., http://www.businessweek.com/magazine/content/07_52/b40640489...). The goal of that program: train the next generation of Google engineers. Based on the evidence, I think your point is exactly wrong.