Hacker News Reader

Comment by ggreer | Hacker News Reader

This sounds pretty cool, but I have a few suggestions:

1. I want to know how much these things cost. Even a line like, "Under $x/hour." would be useful.

2. The blog post says that the memory and compute types have no disk. This confused me at first, but then I saw on http://www.rackspace.com/cloud/servers/onmetal/ that they have 32GB boot drives. You might want to add that to the comparison table in the blog post.

3. Has any thought been given to letting users provision OnMetal servers in specific cabinets? Sometimes users want to have servers next to each other for better network connectivity. Other times, they want them in different cabinets for increased reliability.

Lastly, kudos to Rackspace for building this. Most people don't realize how virtualization can hurt performance. Even without noisy neighbors, a modern Xeon still takes >500 cycles to switch between VM and host.

View on HN

1) I don't think prices have been announced yet.

2) They do have 32GB SATADOMs, but these are intended for boot media and configuration only. They are not intended for anything more than boot media. Think something like an SD Card in a solid-state machine.

3) Yes, I want this. No, we don't do it currently.

FWIW, we didn't build all this. Ironic existed long before we came around, we added a deploy driver and set it all up and exposed it for customers. We'll have a blog up shortly with exactly how we're running this in production.

The coolest part of all this is something nobody has mentioned here yet -- this is all on Open Compute hardware. When I managed servers, I always wanted access to the cool hardware and scale that companies like Facebook and Google had. Hopefully with OnMetal, we're going to give our customers the cool hardware I always wanted.

Edit: As a note, I'm a Racker and I work on the team that built this.

Cool! Can you talk about some unexpected challenge you faced getting this rolled out?

Honestly? Using 512GB of ram is pretty hard. I learned during some of this that you can actually get tmpfs to be CPU limited if you try to write to it too quickly.

> Honestly? Using 512GB of ram is pretty hard.

It's extremely easy. Insert 512GB of data into postgres, there, done.

Not if you are in Oil and Gas :)

Working with IPMI can be tough. We nailed the issues eventually, but there were a few frustrating days of images not booting properly with no explanation. Until we found the right combination of settings, we had issues with flakey BMCs that would stop responding for no apparent reason.

> Think something like an SD Card in a solid-state machine.

Ah, that's still way better than no disk. I was initially worried about having to deal with PXE. I wonder if logging will cause trouble for the SATADOMs. I'm guessing they don't do much in the way of wear-leveling.

I didn't recognize your nick at first, but we worked in the same office. I was part of the Cloudkick acquisition. I left Rackspace in August of 2012 to start Floobits. Small world. :)

I would say if you're running applications in the cloud, you should ship your logs and forgo logging them on disk. Or setup an awesome ES+Logstash cluster using IO nodes :).

You should've left off the last paragraph; I assumed you knew who I was :).

I agree with you about log shipping, but customers are bound to misconfigure their servers and log to the SATADOM.

Here's the blog post, as promised: http://developer.rackspace.com/blog/how-we-run-ironic-and-yo...

Kudos, very cool. Maybe instead of 'no disk' something like 'SSD for boot media, no local bulk storage' would be clearer, as I was a little confused by this as other posters below were. It's also a little interesting that the 'High CPU' option has the fewest cores of the three options... Any comments on that?

It's low on other resources so it's probably lowest cost per CPU.

That's basically correct, we were trying to balance a few factors:

  - Total cost
  - Cost per core
  - Clock speed (this actually matters a lot for, say, a Python webapp)

We chose to run only a single CPU in this box to keep the total cost of the box low. This slightly increase the cost per core (switch ports, the chassis, rack space, etc being basically fixed overhead per node), but by selling in smaller increments we think it we can keep the total cost lower for most users.

We'll be gathering feedback on the resource balance, and iterating as we go.

Yup, this is what I like about Docker (I'm sure there's some overhead as well), application isolation with out paying for it seriously in terms of performance.

That said you can pull a lot of performance out of a modern 2U server, 24 cores and 256 GB of RAM can be had pretty cheaply these days, pair it with 8 SSDs and you've got a real workhorse.

Even a traditional MySQL/Postgresql DB can maintain upwards of 140,000 tps on hardware like that.

And you can probably rack that for order $11k

Or rent it from EC2 for a month.

Reply on news.ycombinator.com