sllrpr

> First, let's show that I can write a job that can produce more data that it inputs. I have a map of user to score, and want to compute pair wise summed scores for every user pair. This is clearly O(n^2) outputs. Computing pair wise scores is a common algorithm for recommender systems (I realize in practice you generally do not compute the entire space because it will be too slow. However your output will be closer to n^2 than n, ie, it will be much larger than your input.)

But that's exactly it - you admit yourself that this is a contrived example! For all but trivial numbers of users N^2 storage requirements would be impractical for any system, no programmer in their right mind would take such an approach.

And this is the point that you seem to be missing: the question is not whether you can come up with some contrived scenario where there is vastly more intermediate data than input data, but when you can come up with a realistic practical example - and apparently you cannot.

> A simpler example is "I want to compute aggregations on all the fields my log file."

In this case the amount of intermediate data is no greater than the amount of output data.

> But again, the point you keep missing and conveniently ignore when you quote my posts is that the point of map reduce on Aws is not about data locality (obviously) but about downstream flexibility. I can run 100 jobs, in parallel, on 10k machines, and output much more data than I input, without running a cluster of my own and I get to pay by the hour. I am isolated from other devs and spin the machines down when finished.

These are benefits of EC2, not of map-reduce. The critique was specific to map-reduce, not of the entire principle of on-demand computing.

> If you buy the argument that this is a useful feature (as any EMR customer would attest too)

Strawman. You're attempting to conflate the general benefits of on-demand computing with the specific benefits of EMR. The author was criticizing EMR in particular, not the entire principle of on-demand computing.

> than this too is a separate more pragmatic reason why the author has no idea what he is taking about when he says all EMR customers are a cargo cult.

And yet you still haven't been able to cite any realistic, non-contrived examples of where EMR is the appropriate tool for the job.

> it was an example of one of several reasons running map reduce jobs on the AWS cloud have nothing to do with the amount input data you are moving around

It would be an example if you had backed up your assertion that collaborative filtering required large amounts of intermediate data, but apparently you are unwilling or, more likely, unable to do this.

> I am not going to go off into even more detail about specific jobs I run daily that generate a large amount of itermediate data because unless I paste the source code in this thread and write a paper on it I assume you won't believe me that there is in fact in the space of "all map reduce jobs" jobs that can generate more data than they input.

Even more detail? You haven't given me any detail! You've yet to give me a single example of a practical situation where a task involves much larger amounts of intermediate data than it's input data. I'm asking you to back up your argument, I'm not asking for access to your source code.

> If you write a trivial map reduce job using cascading that has 10 reducers and each reduce step shuffles the data on a different grouping key you will find that Hadoop alone is generating more data than you input

If it's so trivial, why can't you give me a single practical use-case?

> In practice, the cost of uploading your data to s3 is a footnote compared to the computational flexibility and use cases that become possible once you are able to run arbitrary tranformations on that data via EMR

Yes, apparently so many use-cases that you can't provide a single example of one!

> Collaborative filtering.

What collaborative filtering algorithm are you using that requires terabytes of intermediate storage for gigabytes of input data?

I'm familiar with most approaches to CF (SVD, gradient descent, etc) and I can't think of any that require large amounts of intermediate storage.

> By and large scratch data ends up being much, much larger than the original inputs, if for no other reason than that needed during the shuffle/sort stage

I can't think of a single practical situation where you couldn't do your sorting online as you progress through the data. Again, the overhead of moving the data to-and-from S3 would be greater than processing the data locally (unless Amazon's LAN is faster than a SATA bus, which is unlikely).

> The author sounds like someone who may have read the academic papers and a few books but hasn't used these tools in practice.

You keep attacking the author in various ad hominem ways, yet you haven't yet provided a single uncontrived example of the small input data, large intermediate data scenario that your argument relies upon.

> The point is that small inputs can have large intermediate scratch datasets and even smaller outputs.

Perhaps in some rare circumstances (although not the one you cited), however most people use map reduce for aggregation of one form or another, which doesn't require vast amounts of intermediate data unless you are being deliberately inefficient.

> Even more to the point is that being able to do scalable transformations like computing top N CTR on a lot of data with little regard to available computing/network/disk resources is the reason why you would copy your input data to EC2 for processing.

Actually you'd copy it to S3 for processing, and then it would need to be downloaded into EC2 (unless you want to leave your EC2 instances running, which you won't unless you have a large number of shares in Amazon). It's hard to imagine situations where it is faster to move the data across Amazon's LAN, than to simply process it on the machine it's already on.

> If the author has a point to make he failed to do so beyond making himself look like someone who enjoys labeling things he doesn't understand as a "cargo cult."

The author looks like someone pointing out that the original purpose of map-reduce is that you do your computations where your data is, and that moving your data so that you can do map-reduce on it misses the point. The author is correct.

You might have a stronger argument if you could show some common non-contrived situations where there would be a relatively small amount of input data but vast amounts of intermediate data. You haven't yet.

If the best way you can think of to determine the 10 most popular search queries and their click-through rates involves terabytes of intermediate data for only 10GB of input data, then you're doing something very wrong.

Ugh, I thought I'd answered this but looks like HN lost my response :-(

> How big is the pool of variables it can choose from?

Currently it is just: screenWidth/Height, browserWidth/Height, browser, browserVersion, os, referrer, city, region, country.

It picks the variables based on how well they partition the datasets into diverse sub-groups. This is the code that does that job:

https://github.com/sanity/quickdt/blob/master/src/main/java/...

> It guessed wrong for me but if the 'ExampleCount' value is accurate, you are dealing with a paucity of data.

Yes, you guys are the guinea pigs, sorry about that :-) We'll need a lot more data before it is really going to get interesting.

> Will it look for more decision nodes to add or are you stuck with region and browserHeight?

There are quite a few more, I pasted them in another comment. You can also see them if you "view source" on that page.

> If not, how certain are you that you have the right data to accurately model the problem?

I'm not, the fun of this is seeing how well people's favorite pets correlate with information their browsers give up automatically.

It figures it out on its own.

You can see the entire tree at http://cord-sa.appspot.com/dumpDT

Give it a chance, it only has a small dataset to work with so-far.

I'm happy to answer any questions - this is based on the QuickDT library (see https://github.com/sanity/quickdt).

Many would argue that this is a flaw of C.

Thanks, good feedback.

Actually, after seeking a lot of advice I'm starting to think of the deal differently - I'll update the story so everyone can benefit.

They have technology and customers.

I'm talking to a good lawyer today, they already have the LoI.

We don't really need the money, we each have other incomes that we're comfortable on. This was a speculative side-project.

No, I don't think they've signed an LoI with the other company yet.

Part of me wants to say that is a paranoid view, but I must confess that the degree to which the purchase price ramps up so quickly is a source of concern for me.

$2M to $6M is a dramatic difference. It's plausible to say that they're trying to do a deal that sounds like $6M but is, in reality, $2M.

I'm not sure how that could happen. We have the source code, if they renege on the agreement then we just go do something else with it and sue them if they continue to use it.

Fear not, a lawyer will review the deal before anything is signed.

This may sound arrogant, but I don't think they could build a clone. Our stuff is pretty sophisticated and non-obvious.

Ugh, I'm trying to keep this anonymous-ish.

May I ask why you'd like to know?

> What they'll be looking for however is a significant cut % of the deal so that is what you have to think about next

The thing is that I've got a termsheet on the table that doesn't look half bad. What is my RoI on engaging someone like this if they are going to take a significant portion of this deal?

Asking for advice is not an assurance that I'll accept it uncritically.

If you can't or won't back-up what you are saying then you should probably refrain from giving it in the first place.

Indeed, that is my general view on advice. Advice that can't survive a bit of "picking apart" isn't worth much.

Yeah, I suspected it might be artificial pressure, but the pressure is only to sign an LoI, and their justification is plausible (we need to cut the other deal loose).

After signing the LoI there is 45 days to complete the deal during which everyone will do due-diligence.

Yes, perhaps, if it's all going well enough. Good point, I'll have to think on that.

Sorry, some more:

> Other issues: - You get paid more if they take longer to pay => You're going to work for a company that you hope doesn't blow up tomorrow (because you get paid more if they blow up in 4 years). => This is a dangerous spot to be in; your incentives are not aligned with theirs.

Yes, I did point it out to them. Their response was a polite version of "that's our problem".

> - It might be tempting to think that the acquirer could pay you with the proceeds of their next round even if their business does not take off. But $6M in short-term debt will likely scare off VCs who might fund them in their next round...

I think if the business doesn't take off, we get our technology back and we keep whatever they've paid us to-date. It doesn't seem like a terribly bad worst-case-scenario.

I think you make a good point regarding taking the risk of their business, without getting compensated like an equity holder. I may be able to use that to negotiate for some equity as part of the deal.

The risk to us is relatively limited though, as we get the technology back if they can't pay for it. The main risk is that we won't have made progress on it during that time, but that could have happened regardless (given it's experimental nature, we didn't really have much of an idea how we would market it).

> Any company that doesn't have cash, and is struggling for survival, will tell you that their revenue will explode "next year".

They aren't struggling for survival, where did you get that idea?

> In which case, you take the risk of losing your product and your revenue.

Nope, if they don't pay the agreed amount we get the product back.

> I'd ask for money up front, like 30% of the total, to see their reaction and then negociate the financing.

I might seek more up-front, but I can see why that would be tricky from their side.

> Why is it "unlikely" that they can pay the $2M over a year?

Because this will be a new product for them, they expect it will take several months before they start to make any revenue with it.

> Why such an extreme discount? I'd start digging there.

Yeah, it seems like a big jump. I asked if it could be monthly pro-rata and they agreed to that.

> Is their business viable for 3 years?

I think so, or they'll get acquired.

> Why don't you to ask them to borrow $5M, pay you that amount now, then let them pay off the loan over 3 years? At 12.5% interest, total payments come to about $6M for a 3 year loan term.

Because they don't know that they'll be able to sell the product that they'll be building around my product. This way if they can't they can cut their losses without being on the hook for the entire amount.

I don't see that I'm taking much of a risk, if they terminate the agreement we get to keep whatever we've been paid to-date.

Not trying to show off, I mean, this is a throwaway account FFS!

I'm posting it here because I was told to: http://www.reddit.com/r/startups/comments/loylh/need_advice_...

> what advice can anyone here offer without knowing the exact details of the deal, doing a little research on the company, reading the loi, etc? i seems obvious that the only route of action to take would be to talk to a lawyer (or possibly two) that can advise you.

Just because I'm asking you lot doesn't mean that I won't also seek advice elsewhere.