squarecog

How do you imagine it being involved?

That's an interesting thought, but rather than focusing on write vs read balance, I would claim that OLAP and OLTP are most distinguished by the nature of queries they need to support.

OLAP is characterized by fairly low-volume aggregation queries that touch very large volumes of data ("what fraction of tweets in English come from non-english speaking countries?"). Sure, the ingest is large, but it tends to be batch and has fairly loose real-time (in the EE sense) requirements. What makes OLAP hard is the sheer volume of data from which an aggregation must be calculated.

OLTP is characterized by very selective queries with much tighter real-time bounds ("what are the last 20 things that the 100 people I follow said?"). The overall size of the dataset might even be the same, but each individual query needs fast access to a tiny fraction of the dataset. In many applications, this is accompanied by very high QPS and in Twitter's case, extremely high write volume.

It's not the data coming in that makes things hard. It's the fanout of messages into their subscribers' timelines.

How do you keep the message queue from unbounded growth?

Which CrossFit gym in SOMA are you referring to? (Mine is awesome, too. Just wondering if it's the same one.)

Relevant auto-complete out of the box is a killer feature. Feels and works so much better than the Tumblr defaults, at least based on the reference implementations. Nice job @swiftype.

No -- this is how they weeded out prima donnas.

450 Megs? Use pig -x local :)

50 Megs? How in the world are you getting that? I just checked trunk, and it's 3.1. Granted, Cascading comes in at about half a meg for 2.0, but this 50 meg figure is just silly.

Our tests take seconds, and can run from eclipse (we use Pig's local mode).

You sank 3 days into modifying internals of PigUnit instead of just parametrizing your loader statements?

And Twitter.

We unit test our pig scripts, it's pretty straightforward given MockStorage class we have contributed (you can find it in pig trunk). Granted, we've long been separating load statements from actual logic, which allows us to fairly easily mock up data to feed into the Pig flows. It would be harder if your loads and flows are in the same file.

We don't use Azkaban or Oozie at Twitter, so we are unlikely to do that. But we'll happily take a good pull request. I think the LinkedIn guys are interested.

JSON is a good choice if you have very tight coupling between producers and consumers of your logs.

Twitter (disclosure: I manage/was early engineer on the analytics infra team) used Json initially, and it quickly turned into a mess because JSON does not enforce schemas or type safety. There is no way for your consumers to find out what you are logging, other than to sample records. This makes evolving logging, deprecating fields, and doing sanity checking very complex (you basically have to build a separate metadata discovery system).

Use Thrift or Avro. Avro is extra nice because not only does it have a schema, it keeps the schema together with the data; however, the support for it is not as mature/wide. It's improving fast, though. Thrift, and, I think, Avro, have JSON protocols if you really love JSON -- you get the human-readable debugging mode, the metadata, the type safety, and compact binary representation.. happiness ensues.

You are doing regex matching in the Cascading code, but splitting on a character in the pangool code. The latter is obviously much faster. I don't know that that's the reason for the difference you observe, but it certainly can't hurt to fix that and make the user-supplied code more comparable.

Most of the "forks" are forks from Twitter employees' personal repos that were created before things got consolidated under Twitter's account. A few aren't and are called out as such. Which ones are missing proper attribution?

Disclaimer 1: I work for Twitter. Disclaimer 2: Not a twitter spokesperson, opinions are my own.

I don't understand how so many people are missing the point demonstrated by http://www.focusontheuser.org/. It uses information available from Google search to inject twitter, facebook, etc results into the "your world (as long as it's google+)" module. The ranking it uses is Google's ranking of the relevant results. No special API access. No back-end deals. No money. It's stuff that's available to Google and that Google already has and has already ranked.

If they choose to dilute the worth of their search engine by injecting less-than-relevant (according to their own relevance algorithm) results in order to attempt to grow Plus (presumably hoping that'll ultimately increase the revenue to offset the losses on the search engine side of the business), they are perfectly within their rights to do so. Just don't buy their "oh we'd be happy to show more stuff but boo-hoo we don't have the data" line -- they totally have the data, and it doesn't require sweetheart deals. The only thing they can't do is grab your personal social graph and re-rank according to it. So they use something they know is sub-par (Plus's currently very partial graph) and don't bother dialing in relevance signals against their own best-in-the-world search engine.

Nothing is wrong with byte buffers. Use them where appropriate. The advice is to stop when you find yourself implementing a full-blown memory manager / quasi-malloc in user code on top of byte buffers...

The context isn't clear from these notes.

Full context as explained in the talk: used to have stop-the-world GC for 2 minutes every hour. After implementing bytebuffer-based slab allocation, this is only several seconds, and once every three days. Service runs on 200 nodes, with redundancies. Kicking the process on one of them, in a slow roll that finishes in under 3 days, works around the unresponsiveness window (planned shutdown easier to manage than an unpredictable pause).

It's totally a workaround and not a solution. Atilla follows up this example with an anecdote of talking to Oracle folks about when are they going to have true pauseless GC, and they responded with "not that big an issue, really, everyone finds a workaround..." So, this is an example of a workaround. A pretty good one once you realize it's not "the one" machine, it's "some one" machine out of 200.

Except, supermodels don't eat.

Couple hours later, Hogan is faster than Handlebars: http://jsperf.com/t-bench2/7

I would argue that any time you put "just" and "terabytes" next to each other, you are heading for big problems to go with your big insights :). Schema-less is great.. until you can't find stuff and your data is full of inconsistencies.

The HBase integration with Pig is pretty good (disclaimer: I wrote a bunch of it, and use it on a daily basis). The only thing is that you need to create the table and set up column families yourself. The mongo driver Russel demoes automatically creates a table which may or may not be a good thing. Also, he didn't actually say anything about scalability except for linkbaiting in his title :).

Where did you get that this is being marketed for production? Did you just assume that because the site looks nice?

Handy tip: languages that are marketed for use in the real world tend to advertise actual uses in the real world.

CoffeeScript. CS. JavaScript. JS.

(I am one of the authors of Crane) The thing that happened with Crane is that it grew a lot of Twitter-specific cruft and separating that from the generally-useful bits is a serious undertaking -- and I'm not sure how much useful stuff would be left once we cut the twittery bits out. Now and then I chat with the Sqoop folks about how we can merge the two but that path isn't clear either.. so it's in a bit of a limbo, as far as being open-sourced. Sorry about that.

Detailed slides here (pdf): http://www.lucenerevolution.org/sites/default/files/Lucene%2... Video: http://vimeo.com/16063395

Right. Let Google's failure to make bucketloads of money be a warning to other firms that might have similarly laughable ideas!

What a waste, that company really looked like it might be profitable one day.

In addition to the criticisms you address in your addendum (women's perspective is important, and "works or not" is too simplistic) consider this: by creating and perpetuating environment that is unfriendly to women, we are reducing the talent pool from which we draw programmers by sightly more than 50%. Even if there was no such thing as "a woman's perspective" (depends on the woman/man in question, I suppose), it is highly inefficient to limit your pool this way. In my opinion this is NOT, in fact, self-selection, not to the degree this imbalance is getting to. I've seen multiple times in the universities I've attended the difference between how things get explained to women and men -- many instructors and TAs, for example, will quickly and subconsciously drop into condescension and use things like cooking analogies when talking to female students. I've heard programmers of all talent levels apply gender-based value judgments to women attempting to enter the field. The worst are "women are bad at logic" -- and there are many who really believe that -- but "you are a woman so you will do the UI as you must be good at visual stuff" is also sexist.