Forrst Blog: How we deploy new features on Forrst | Hacker News Reader

Forrst Blog: How we deploy new features on Forrst

blog.forrst.com

54 pointsschapman62315 comments

I really don't see Flickr's (or Forrst's) approach a "trick" or feature. I think most deployment techniques come out of a desire NOT to deploy like Flickr(or Forrst).

This idea is also well explained here: http://news.ycombinator.com/item?id=1463751

Where can I get an invite? I applied on the site but haven't heard anything and I can't wait any more!!!!!

e-mail me mjhunter [at] gmail

How do you modify the buckets? Do you have to tweak a config file and push it out to your servers?

Right now, yes. Deploys happen frequently enough and are sufficiently fast that I can get something disabled globally in under 30 seconds. The next iteration of this stuff I'm building now; goals are standalone library, more DRY, and some kind of simple web UI where I can do the global enable/disable actions. I'll likely stick to manipulating buckets in config files for the foreseeable future; it's being able to mass-enable/disable that I need to refine.

I've started using Redis key/set operations for things like this at GitHub. The nice thing is that these techniques are all simple, and it's easy to modify to fit your evolving needs.

Yeah, I definitely considered using Redis too; I'd be able to leverage web UI for even stuff like adding/removing users from buckets. How are you handling with persistence? I presume backing up the dump.rdb file frequently? I'd be a shame to lose any of the data (though I'm probably being overly paranoid)

We were already using Redis and had an existing backup plan. Though if you're using Redis solely for feature rollout, you can likely get away with it failing gracefully in case the server is down.

Using a combo of the append-only file[1] and a redis slave[2] you'll get about the best durability one can ask for in today's world (assuming you're aware of the gotchas with regard to expiring keys — an edge case).

[1] http://code.google.com/p/redis/wiki/AppendOnlyFileHowto [2] http://code.google.com/p/redis/wiki/ReplicationHowto

We are using Redis as well. The C code is small, compilable anywhere, and has very small memory footprint when started fresh.

Redis has replaced Sphinx as our tag indexer (thanks to its ordered sort) for more than 2 months now. It has never been down.

Backup procedure is the same as our MySQL's, tgz the .rdb file to S3 daily.

Wouldn't this make debugging and (manual) testing so much more difficult?

How so? I'm not sure I follow...

Not really. You just add extra tests for your new functionality as usual. If you're replacing old functionality, you'll have duplicated tests until the feature is rolled out for everyone.

I think the key is being able to tell if bug reports are related to an activated feature. For instance, our error tracker tells us which user account triggered the error. From there, we can see what new features they have enabled, and factor that into our debugging.

Feature flags are essentially identical to operating A/B tests in this respect, right? I have a magic button which logs me in as a target user, and it reproduces exactly what they would see, including the results of A/B tests. This helps to reproduce unanticipated interactions between tests. (This has only happened maybe twice, but they were doozies.)

Reply on news.ycombinator.com