Hacker News Reader

Comment by andr | Hacker News Reader

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

View on HN

I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark.

Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market.

With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience.

As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.

That's a great idea.

It's alright as an idea, might not be too great as a business depending on the demand.

You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny.

If anyone once to give it a try I'd like to see how it turns out for them.

If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop.

And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.

Deleted comment

Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.

Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs.

Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before.

I just launched the SAME JOB in 15 seconds.

You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity)

I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.

Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.

There's no way to automatically kill the cluster after the job is done? There definitely should be.

I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).

Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown.

Elastic Reduce does that for you.

Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.

There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done.

http://groups.google.com/group/ec2ubuntu/browse_thread/threa...

Reply on news.ycombinator.com