Hacker News Reader

jaylevitton Scaling lessons learned at Dropbox, part 1

I think (I assume) his point was that when you have a complex system that's growing rapidly, it's difficult to predict what the next breaking point will be. Building in extra load gives you a window of opportunity: You discover the breaking point, turn off the extra load, re-engineer, and turn back on the canary so you can find the next breaking point.

In theory, you'd think you can do load-testing and simulations and capacity planning, and find these breaking points ahead of time. In practice, it's not always feasible, and this seems like a simple-enough hack that gets you much of the way there.

jaylevitton Aaron Winborn: Special Needs Trust

See especially Jamie's TEDx talk. "Wouldn’t it be great if the technology we used to take care of ourselves was as good as the technology we use to make money?"

http://www.ted.com/talks/jamie_heywood_the_big_idea_my_broth...

And to bring it back full circle to HN: Once original Basecamp sunsets, we think PatientsLikeMe has the oldest continuously-running Rails app; the oldest commits are from 2005, running Rails 0.5.

jaylevitton Use OSX Finder Quicklook (Spacebar) to preview all plain text files

I bet RCDefaultApp will work:

http://www.rubicode.com/Software/RCDefaultApp/

jaylevitton City of San Francisco Will No Longer Buy Apple Products

Very few scissors store company-confidential data.

jaylevitton On Configuration

GAH! See, I just slipped up and showed my age. I'm thinking of the Stratus VOS filesystem, which did change the modifier when you edited a file. You're right; UNIX/Linux files have an owner, which doesn't change on modification. Which shoots a big blazing hole in my argument. You can back into who modified what if files were changed in a big batch, and they also created a few new files - but you could also back into that via login times.

Since those days, it's less that I've seen the individual-user system work at scale - it's that I've seen the single role account fail, repeatedly. I've managed to avoid permissions hell with a few well-chosen supplemental groups, but maybe I'm too optimistic that what used to work in one environment will work in another. Ask me in six months. :)

jaylevitton On Configuration

I take it you have not worked in such an environment before

Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base).

I have worked with some amazing people, at all scales, and I have never seen a deployment that managed to completely avoid manual logins. No, you aren't logging into a few hundred hosts manually (well, there was this one shop... but let's not hold that up as exemplary). But you're probably logging into a few to test things out manually before you decide what your automated deployment script will be rolling out.

Even when you do deploy a fix, with a large sysadmin team it's good to know WHO exactly ran that one-off deployment script. When I said "auditing", I wasn't even thinking of PCI, or auditing against malicious actions of any kind - just simple troubleshooting forensics. But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? (I just spent the last day trying to figure out who edited a script. Why, "postgres" did.. of course it did.)

Should you aim to do these manual config changes in the test lab first? Of course. Do you always succeed? No. Some things only get tested in the big lab. Reality intervenes. I try to plan for it.

Mind you, I'm arguing this partly to see if I can be talked out of it. I think I believe it, but I've been wrong about way bigger things. If you've achieved my automation nirvana, and you roll things out to a few thousand servers, never manually, and never wondering who did it, tell me how it works.

jaylevitton Keyless BMW cars prove to be very easy to steal

Are you sure you linked to the right thread? I didn't see any discussion of the legal aspect at all - other than "can we sue BMW?"

jaylevitton On Configuration

In single-server deployments, you're logging in because you're the only employee.

In ten-server deployments, you're logging in because every machine is trying to take on a new role or aspect and you're still learning how to automate that.

In thousand-server deployments you're logging in because the latest batch of Seagate drives has a statistically significant failure rate and you need to try some experimental firmware they wrote for you.

When do you get "serious"?

I don't always log into my production boxes... but when I do, I log in as 'jay'.

jaylevitton Keyless BMW cars prove to be very easy to steal

Are you speaking from knowledge of the anti-competition legislation, or just from the summaries we've seen upthread? It strikes me that we're taking a car blog's throwaway description of the law as a presumed engineering constraint.

jaylevitton Trello is now half million strong

Sure, if I got a designer tote bag.

jaylevitton Apple Releases Final Mac OS X Mountain Lion to Developers

Frankly, if the only punch Mountain Lion packs is "breaks way less", I am first in line.

jaylevitton On Configuration

You're saying it's better for sysadmins to all log in as "prod" or "root"? How do you figure?

jaylevitton Bringing Craigslist Back

What? Cite? A quick Google seems to show the opposite.

jaylevitton About the future of Thunderbird

That's some serious newspeak, Mozilla...

"Thunderbird is not dead. We have announced a change in the way we develop new features for Thunderbird."

... in the sense of "not doing that anymore".

jaylevitton On Configuration

No, what this means for DevOps is that you'd better create a foolproof method to insert your dotfiles into the distribution - and that it's more important than ever to argue for individual user accounts, not role accounts, on production machines. For security and logging, of course.

Love, DevOps

jaylevitton So, That’s It For Thunderbird

Seems unlikely; Scott MacGregor left Mozilla to launch Postbox.

jaylevitton StackOverflow Man Remakes Net One Answer at a TIme

Is it possible that most closed questions are "too subjective", but that most unfairly closed questions are "off-topic"?

As someone who's had a few questions shunted from SE site to SE site, being closed at every step, I think SE would do well to emphasize migration over closure. Having a question closed is a real downer, the opposite of all the SE game mechanics. And it's one of the oldest principles of UX: Don't tell me what I should have done. Just do it for me.

jaylevitton Summary of June 29 AWS Service Event in US-East

To my ears - and maybe this is just wishful hearing - it sounded like they were very, VERY strongly pointing the finger at a certain unnamed generator manufacturer, but doing so in a way that incurred no legal liability.

That manufacturer is probably flying every single C-level exec out to the US-East data center, over the July 4th holiday, to personally disassemble the generator, polish each screw, and carefully put it all back together while singing an a cappella version of "Bohemian Rhapsody", including vocal percussion.

And if they do it to Amazon's satisfaction, Amazon has hinted that they might decide not to out them to the rest of the world. That's called leverage.

jaylevitton Sortfolio lives

I think it's unfair to expect any of us on HN to understand this metaphor.

jaylevitton MVC is dead, it's time to MOVE on

Are you volunteering to help push through wahnfrieden's REST changes, then?

jaylevitton Facebook's e-mail debacle: One 'bug' fix, but rollback impossible

There's no reason to assume that the old addresses in people's address books were the same ones Facebook had as primaries. If my co-worker was in my phone as tom@patientslikeme.com, but signed up for Facebook as tom@gmail.com, and Facebook tries to roll back? Data loss still, and possibly even a privacy leak.

jaylevitton Privilege escalation vulnerability on 64-bit Intel CPU hardware

AWS says no:

http://aws.amazon.com/security/security-bulletins/xen-securi...

jaylevitton Books Every Entrepreneur Should Read

_The Halo Effect_, Phil Rozenzweig: http://www.amazon.com/The-Halo-Effect-Business-Delusions/dp/...

This revisits books from earlier in the decade (like _Good to Great_), and shows that many of the profiled companies failed. It's a great way to learn how to read entrepreneurship books; many theories are based on "Intel is successful; Intel does X; therefore doing X will make you successful".

In general, go read books from ten years ago and see if any of the predictions held up. Likewise, I recommend _Founders at Work_ because so many of these (currently) successful companies were launched with opposing philosophies, and all of them worked. Open plan? Works! Offices? Also works! Deep funding? Works! Bootstrapping? Also works!

jaylevitton My boss decided to add a “person to blame” field to every bug report

You're right. I feel sheepish and proud and part of a bigger effort, but I do not feel nothing.

jaylevitton My boss decided to add a “person to blame” field to every bug report

Really? My first major software release took the AOL mail system down for 34 hours. My later work kept it up for 8 years. I consider both equal fodder for bragging, self-immolation, and war stories over drinks. I couldn't tell you which taught me more.

jaylevitton If malware authors ever learn how to spell we're all screwed

This assumes that users are computer-savvy enough to (a) expect an unspoofable dialog at all, and (b) know which parts are unspoofable and how they should look.

We tried this with "AOL Certified Mail", which had an unspoofable official chrome, and I don't remember any serious drop in phishing.

jaylevitton If malware authors ever learn how to spell we're all screwed

SiteKey is also trivially vulnerable to the attack known as "I bet you didn't remember that this page should show you a SiteKey."

jaylevitton If malware authors ever learn how to spell we're all screwed

This was known 20 years ago; see e.g. the seminal UI book "About Face".

jaylevitton Amazon EC2 and RDS in US-EAST zone down

According to this calculation (which attempted to probe all the racks in EC2), over 70% of EC2 lives in us-east.

http://huanliu.wordpress.com/2012/03/13/amazon-data-center-s...

jaylevitton Bob Mansfield, Apple's SVP of Hardware Engineering, to Retire

Am I the only one hoping his retirement leads to a second career doing voiceover work?