Hacker News Reader

Comment by jaydub | Hacker News Reader

This past summer I worked as an intern for a big company. On the first day of orientation the head of the legal compliance department said very flatly to us that any individual could do far more harm to the firm than good.

This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.

Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).

View on HN

For large companies with a valuable reputation, that's almost guaranteed to be true. Most people won't generate a million dollars worth of value in a given year, but nearly everyone could do that amount of damage to their company's reputation in just a few minutes.

Sure, yes, if e.g. the customers' health or safety was at risk. But I don't think the average bug in a web app would damage a company's reputation significantly. GMail occasionally shows me a message saying "Oops, an error occurred." It doesn't make me think any less of Google.

Imagine if, say, an Amazon customer service rep actually yelled, screamed, and abused a customer calling about his order.

A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.

Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.

I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.

You aren't a typical non-early-adopter user. I've had people make loud, public complaints about such trivial matters as a new logo being too tall, or a text box being too wide, or an optional WYSIWYG editor feature ruining the "text-only flavor of the community". These on websites far smaller than GMail.

Look at some of the major Web2.0 kerfuffles in recent years. Off the top of my head, I can think of:

- The HD/DVD mutiny on Digg

- Public suicides on both JoelOnSoftware and Justin.TV

- Reddit storing passwords in cleartext and them getting stolen off a laptop

- Ariel Waldman and the Twitter harassment fiasco

- The Flickr censorship debate here: http://www.flickr.com/help/forum/40074/page3/

Yeah, you could argue that those services are all still around, so obviously they haven't been hurt too much. How much management time was wasted in dealing with them? How many non-users decided not to become users because of something they heard 3rd-hand about how it's a terrible company?

(And I don't think the solution is to never innovate. I can understand why a middle manager at a big company would think so, though - a PR disaster is by definition public and disastrous. In the short run, it always makes more sense to not mess with success, it's just that this doesn't lead to success in the long run.)

None of these were the kind of bug that comes from releasing code with insufficient testing. The only one that even involved code was the Reddit password problem, and that was more a design mistake than a bug.

A culture of testing is a lot more than just verifying functionality. Code reviews with experienced engineers would have caught that design flaw before it was released. Or made it a high priority to fix.

I've dealt with the security teams at large companies. They end up implementing sometimes draconian policies, but it seems that programmers refuse to learn to write secure code any other way. I have to admit that, if you consider the large company as being a legitimate entity, they're doing legitimate work.

Privacy breach could harm a company a lot. Japan Network Security Association reported that, in 2007, total of 30 million personal information records are breached and estimated 22 billion dollars for damage/compensation in Japan.

Most of these incidents caused by employees taking out the data, but there are some cases caused by sloppy web app.

(I could find only Japanese version of the report: http://www.jnsa.org/result/2007/pol/incident/index.html )

The kind of bug big companies are concerned about is the kind that causes google.com to be down for 15 minutes. That's a significant amount of revenue. Or the sort of bug that causes a release of an infrastructure component to be rolled back, causing more work for every project touched by that release. It isn't the famous sort of bug, these don't get picked up by the news, it's the sort of bug that nobody really notices but costs the organization a lot of money or work.

If gmail deleted all your mail it might, though.

Well, what if this bug happened to allow content which was not-safe-for-children onto the front page of the most heavily trafficked web page in the world, widely promoted to Middle America as being G-rated? I speak purely hypothetically of course.

If the error was a security vulnerability in Lotus notes that resulted in all of GM's confidential email being uploaded to thepiratebay I think IBM's reputation would take a pretty serious hit.

And this is an error that one unchecked programmer would be perfectly able to make if there were no checks in place.

The average billion dollar company is not in the web app business.

Most people might not generate a million in value per year, but some people can generate a million in value per day.

The flip side is that if an employee does generate that much income in one year, a large company isn't set up to reward it.

Case in point: I work for a $3Bn company and a year ago I developed a feature that saved the business ~$2M that year. We know the amount because when the problem raised its head we called together a number of people to calculate what the cost would be and estimates were between $1.5M - $3.5M if we couldn't find a solution. Not loss of revenue, but we would have actually had to pay out that much in support costs. The problem was highly visible across the organization and I was able to find a way to solve the it in a way that was completely invisible to the customer, implemented it and released to the field. My manager looked good, his manager looked good. I got an "average" performance review. I was pissed and to this day I haven't regained my motivation to do that caliber of work again. I wonder if I ever will without leaving!

Reply on news.ycombinator.com