TimothyFitz

UI is an interesting problem.

The ultimate solution is to have business metrics drive your UI changes, usually in the form of an A/B test. Then you have a clear winner. This A/B would be run separate from the roll out structure (and indeed, we do LOTS of A/B tests).

Sometimes that's not possible, for a new feature or for content without a clear business metric to evaluate for. Either way we often have someone manually test new UI, so that we're not exposing users to something fundamentally broken. We usually do this by using the existing deploy system, but turning the frontend on only for QA users.

In the end, you do what works and is cheap, and that's usually something slightly different for every project.

I definitely wasn't disappointed. In fact it was refreshing; I didn't realize how hard of a problem other people considered Continuous Deployment (and in fact, how little other people had considered it at all).

I also didn't mean to rag on news.yc, on average the responses here are better than the original posts which is an unparalleled level of quality.

I left the technical issues unspecified in my first post on Continuous Deployment, and the comments on that post already had started discussing the path of solutions we ended up building ourselves!

I still have another post to write about this, because we also ship a native windows client. We ship daily prereleases of it, and roughly biweekly full releases (offered to all users). That's close to, but not quite as impressive, as the update system that google chrome uses. Google definitely still has us beat in certain categories.

Honestly I think it's a gradient.

I'm also one of the developers on a hobby project called http://TIGdb.com (Jeff Lindsay is the other, and has written the majority of the website) We don't have a big Continuous Deploy infrastructure, but we also don't have the users and business requirements of IMVU.

We started with the usual, completely manual deploys and hard-to-setup sandboxes, and have been iterating towards a fully automated setup ever since. The entire time we've been doing this, we've been committing and deploying often. Our users are patient, because we're giving them something they can't get elsewhere and we're giving it to them for free. As we do introduce regressions, we'll post-mortem them (probably using the 5 why's technique) and we'll slowly evolve a system to prevent regressions. If the site is a success, we'll have evolved a world class deploy system. If the site never makes it that big then we won't have wasted time on infrastructure. It's classic lean startup thinking (even though TIGdb is really just a hobby project).

Exactly. Drew Perttula put it better than I'll be able to: "IMHO, manual testing has only two advantages: it’s the easiest thing to [try to] do; and it has a lovely accountability chain. You can always blame the developer, and non-technical people will easily accept that this is the “inevitable cost of software engineering”."

It turns out the system works fine with SQL based alters. We do have to do real work to deploy expensive alters (apply them to standbys, fail over, repeat, or worse) but in general it's cheap to change schemas.

Unfortunately, it's very manually intensive to roll back schema changes, so it's one of the few places where we put old school process in place (a DBA who reviews all schema changes prior to deployment)

You're right. If everyone behaves the way they do with larger deploy cycles even after moving to an extremely short deploy cycle then you would have lots of downtime.

The point wasn't that it solves downtime, but that it causes you to more readily understand sources of downtime. You then have to use that information to make your development and deploy process resilient to those classes of failures. Rinse. Repeat.

For example, at my company we would spin up a new engineer and within about a month they would take out a database (or worse) with a query that would be extremely slow. What did we do to fix the problem? We capped all queries to 20 seconds, after which we would fail the page request. In the best case, a developer pushes a page with a slow query in it, and the page fails enough to cause the revision to roll back. In the worst case, the feature the developer was working on would be broken but the rest of the website would continue functioning.

And yes. yes. yes! You should be catching your problems before they go live. The list of techniques is staggering: unit testing, regression testing, functional testing, fuzz testing, exploratory testing... but those methodologies only go so far. Continuous Deploy goes further.

What we implement is much closer to this suggestion. We move the code out to a couple machines from each of our different web frontend servers. After a minute we compare before and after across numerous metrics (load average, cpu, errors, page failures, etc). If the revision passes, we roll it out to 100% of the cluster and do the same monitoring for 5 minutes.

Originally we intended to put business metrics in these tests, but it turns out we regress on them via code changes rarely and it takes a human to figure out what went wrong. Instead we test business metrics (and lots of other stuff) via nagios, which gives us 1-5 minute sampling frequency, good enough for most of our issues.

I did not cover how you would iterate towards the ideal (instant deploy) and what concessions you might have to make. Our 6 minute deploy is actually quite inefficient, but it's not the bottleneck to our deploy system.

(If you're wondering what our bottleneck is, our automated tests take 9 to 12 minutes despite being spread across 40 machines... Selenium in Internet Explorer is slow.)

I was really surprised to read that every employee had access to profit margin data.

Kudos to Wal-Mart for a level of decentralization and transparency many startups and high tech companies don't even achieve!

Read more about webhooks http://webhooks.pbwiki.com/ and http://www.slideshare.net/progrium/web-hooks-and-the-program...

Micropayments + virtual goods & streaming music rights: http://imvu.com

Yeah, TortoiseGit being hosted in SVN seems ironic. Perhaps they're waiting for it to be stable enough to dogfood before switching to Git?

Damn. If I weren't deliriously happy with where I am, I'd consider applying just because you said "mean level of seriousness"

To contrast, Unununium was intended to use python to make rapid prototyping of novel userland functionality a reality. Pycorn looks like it's primarily a teaching tool. Both are cool conceptually.

I wouldn't mind a placebo downmod. Just to get those feelings out.

This idea has (IMO) a better name, WebHooks, and a standard place for discussion about the issues and tools around the concept: http://webhooks.pbwiki.com/

XMPP unfortunately ends up being the "heavyweight" solution; there's probably a decent argument that WebHooks are to XMPP what REST was to SOAP.

(And don't get me wrong, what I'd really like to see is websites that support RSS, WebHooks and XMPP, each has strengths and can compliment the other)

The functionality is already a jump over everyone else in this space.

Forget graphical issues; do a real usability study. Then start funneling ad traffic to the site and start A/B testing more usable interfaces (aka don't take my word for it, use data!)

But that's exactly the point. You can be MORE lazy. You spend so much more time than you realize testing and re-testing your own code... doing monkey-level QA! Stop doing all that work. Be more lazy. Write your tests first!