pquerna

thanks! Author here, happy to answer any questions...

Amazon does not have a 'baremetal' instance type.

OpenStack 'can', but in practice the baremetal nova driver[0] has many issues, and the newer OpenStack Ironic Project[1] is not yet ready for production workloads.

Additionally, both baremetal and OpenStack Ironic have a focus on 'triple-O', or OpenStack on OpenStack[2] use cases -- that is using these projects to bootstrap an 'undercloud', for the hardware to be used by a higher level virtualized cloud.

[0] - https://wiki.openstack.org/wiki/Baremetal

[1] - https://wiki.openstack.org/wiki/Ironic

[2] - https://wiki.openstack.org/wiki/TripleO

Shipping binaries on Linux is nearly impossible on a long term support basis.

FatELF would of addressed at least some of these problems, but was largely rejected by the larger community of people not shipping proprietary products on Linux: https://icculus.org/fatelf/

Why is this a great lesson?

The test is not repeatable. There was a power outage. Random machine configurations on random network conditions. These are all great ways to stress redis-cluster, which is still young, but they are not repeatable tests.

They are not great tests. Great tests would be a script that can run every 24 hours over a group of servers, simulating or creating similar events and network conditions. Then when a code change is made, you can actually observe if it made things better.

Driving a car across the Sahara Desert can be a good event, but it shouldn't replace Crash Tests in a Lab environment.

Yes, it highly depends on what your foundation "does".

The ASF does not hire developers to work on projects -- it just provides the infrastructure for development to happen. Once you start hiring developers, budgets for foundations go up way too quickly, and once you do that you get into the whole fundraising trap. (Need to raise 2m? Better pay an ED $250,000 to do that, and then they are motivated to get it to $3m, etc)

Cassandra binds to posix_fadvise to do exactly this when writing out new SSTables:

https://github.com/apache/cassandra/blob/trunk/src/java/org/...

Many of my ancestors immigrated to the Americas in the 17th or 18th century.

Best case I have something like 4th or 5th cousins in Europe. Not someone I can just drop into their lives.

You can basically add as many or as few bytes as you like, by abusing the chunk-extension in chunked encoding:

http://tools.ietf.org/html/rfc2616#section-3.6.1

So you could make all http responses round into 128 byte chunks, by appending 1 to 128 bytes at the end of every response.

Effectively it gives you length hiding at an http layer; Still attackable.

Yes, it functionally the same.

The difference is that it is extremely easy to add at an http proxy or load balancer level, and could potentially turn 30 seconds into hours;

I would love a way to figure out the math on how many bytes of random response length changes the number of requests needed?

Has anyone looked at mitigating the attack by changing the behavior of chunked transfer encoding?

Chunked Transfer encoding is basically padding that a server can easily control, without having to change content or behavior of a backend application. A web server could easily insert an order of magnitude more chunks, and randomly place them in the response stream.

I previously wrote about making a downtime page:

http://journal.paul.querna.org/articles/2009/08/24/downtime-...

One thing you'll probably want to do is set 5xx HTTP status codes on your static site, and to try to get things to not cache it as much as possible.. A redirect to a subdomain hosted on s3 makes it more likely something like Google will pick up on it.

Sure, Finance will be the first to adopt, but it actually has pretty big implications in larger internet use cases.

Lets say your application needs <50ms latency, low enough that your users can't notice. Today, you'll something around 10 datacenters in the USA/Canada to achieve this.

If cut that down by 31%, you might only need 6. (or less, this would take some real math to figure out).

But the impact is pretty large in the long run, if it can really cut long haul latencies by 31%.

Rackspace use hundreds of open source projects.

Luvit.io is specifically being used in our Host Monitoring Agent, which is open source and you can find here:

https://github.com/racker/virgo

This is used to collect all kinds of host metrics (CPU, disk, ram, custom scripts, mysql, etc) for the Cloud Monitoring Product.

ObjectRocket provides 2.2.

The High-I/O Quadruple Extra Large starts at around ~$2232 per month for 2tb of SSD storage.

With this kind of pricing you will have a hard time meeting user expectations about price; Even with high multi-tenancy on top, and if you tune up that density too much, you will probably push too much ram, as the instances only have 60.5GB accessible, which with 2tb of disk is pretty unbalanced for many mongodb use cases.

Disclosure: I'm a Racker and worked on this project.

I think this will be a great product for startups -- everyone wants to build products on top of a dynamic environment, but to really do it, you would need to invest in heavy use of things like using Zookeeper -- but Zookeeper has a very high cost to integrate and maintain.

Netflix has released something very similar, Eureka, as open source here:

http://techblog.netflix.com/2012/09/eureka.html

But the Rackspace Service Registry comes with a SaaS model, more client libraries, and is hopefully easier to integrate.

(disclaimer, I work at Rackspace, but not on this product)

FWIW, at least on apache.org, this is intentional:

http://www.apache.org/server-status

Additionally, mod_info is enabled, which lets you read nearly the entire running configuration:

http://www.apache.org/servinfo

Coincidence -- I know the Block Storage team has been working very very hard on this product for a long time :-)

This is incorrect for Apache 2.4: In 2.4 there is async write completion.

SSL behavior is still like the worker MPM, due to design issues in mod_ssl and filter stack. This is unlikely to change until 3.0.

I really want to give a talk about using libuv. It isn't a perfect library, but it is really one of the first to build a portable API for high performance event loops. (libev is great, and on *nix libuv uses it internally, but on Windows it doesn't use IO Completion ports)

https://github.com/joyent/libuv

Author here: Yes, the README is aspirational, its a checklist for myself. I put the STATUS line there for everyone else, but it appears no one actually reads it. (Heck, not one other person on HN actually tried it out, because it sure as heck doesn't work yet).

I hadn't put any publicity out about the code yet, I had just pushed it up to github a few days ago because I wanted to move the code around to some other machines, I didn't expect anyone to pick it up.

I mentioned ASN.1 DER, but I honestly didn't think I should go into a history of XDR or other encodings. I guess I can't skip any history in a blog post....

And during an acquisition GPL'ed software dependencies is at the very least a yellow flag to be fully investigated....

San Francisco, CA – interns, remote, h1b, whatever…..

Rackspace http://www.rackertalent.com/san-francisco

Here at Rackspace in San Francisco we are hiring across the board. We are looking for Python/Django Engineers, Front-End Javascript Engineers, Pyhton/Twisted Engineers, Node.js Engineers, Test Devleopers, DevOps. Project Managers, you name it.

We have an awesome new space in the SOMA district of San Francsico, and we have lots of room!

We are open to hiring interns, remote workers or candidates on H1b visas depending on the role.

We are working on some very exciting cutting edge products that are highly visible and will be products used across our entire organization and be used by all of our customers.

If your interested please feel free to reach out to me <paul.querna@rackspace.com> or Blake Haggerty <Blake.Haggerty@Rackspace.com>.

http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2011/11/17/...

I wrote this up awhile ago for how to do this in pure httpd configuration:

http://journal.paul.querna.org/articles/2009/08/24/downtime-...

They are using TLS to assume that the client authenticates the server is who it says it is -- if you can man in the middle TLS, there are lots of other possible exploits besides this signature one.

Having yet another HTTP Authentication and Signature scheme really isn't adding value to the world.

It's too bad this couldn't model the Keystone token authentication API that OpenStack is doing:

http://keystone.openstack.org/

Or say, heck, don't like OpenStack, even just using OAuth2 would be fine:

http://datatracker.ietf.org/wg/oauth/charter/

It sounds cool, using SSH keys as your API signature, but really it isn't solving a problem that hasn't already been solved.

Maybe.

Protobuffers or Thrift can definitely be smaller in byte size, which does matter somewhat, but JSON right now is just more easily accessible.

I think in a few years, assuming JSON logging really takes off, you'll see people using these other RPC formats in order to optimize and reduce the size of their messages.

I view 'better' RPC formats as a performance optimization of the same basic idea though: Easily add new fields, and use a format designed for computers to consume.

Go's core packages are good.

Go's not-in-core packages are a mixed bag. Lots of "one off" experiment projects. Which is fine, thats where it is in its life. Node.js was in the same place packages wise 18 months ago.

Additionally, at the time we were looking at this, Go was still doing releases every 2 weeks -- this was months before the Go 1.0 plan was even announced: http://blog.golang.org/2011/10/preview-of-go-version-1.html

The state of the spreadsheet wasn't constant.

We played with the weights.

What I posted was just where it was left after a meeting 9 months ago.

Yes, we definitely weight against our feelings of how we failed to employee Twisted Python. This was our experience, are you suggesting we shouldn't consider our experiences when evaluating something?

It really came down to a choice between pursuing a JVM based system, and Node.js.

Yup, we probably did want to do something new. But guess what, its worked. I'm sure we would have haters the other direction if we were posting about how Node.js has failed us, but it hasn't, yet.