Hacker News Reader

Comment by makeitdouble | Hacker News Reader

> insufficient testing

There is some magical belief that people can understand everything whatever the size of the system, and with enough efforts and processes you can guarantee nothing bad will happen 24h after deploying new features.

I actually see this belief as a red flag, as it usually comes either from hubris or misunderstanding of the technical complexity of their stack. Interestingly even if there are exceptional people that actually have a crazy good understanding of everything, the org is probably heavily relying on them, and when they're gone the culture left won't be kind to those who aren't exceptional.

PS: one can assume that whatever happens 24h after deploy on a friday is the on-call team's problem, but that's not an attitude I'd value in a company.

View on HN

Alternatively you can look at it from cost/benefit point of view. Cost of testing grows exponentially as you approach the 100% certainty. You can say "it's probably fine" with an okay test coverage and a few minutes of poking around in a staging environment. But to say "it's 100% definitely fine" you need to fully test every last bit of the whole system, with real data, in realistic environment and load.

If the project is launching a space ship, then investment in every possible testing method is probably worth it. But when maintaining a blog, you can keep costs down if it can wait until Monday.

Yes. Also, even actual space ship launches are done when there is the largest margin for error, and so the best chance of success. Not "whenever it's ready"

A literal example of this is that US Space Shuttles are never launched nor land (except if there was an emergency) on the last and first weeks of a year, and until 2007/2008 (where the software was meticulously tested and fixed in summer of 2007) missions are scheduled to avoid December 31 and January 1 in space. This is a precaution against year-end rollover (YERO) bugs (https://abcnews.go.com/Technology/story?id=2699091&page=1).

Yes, the attitude is arrogant and demonstrates hubris IMO. Next up, no thought to how to back out changes. Even if your software works perfectly as intended, you can still need to back out because someone else was not prepared or could not adapt to the change quickly enough. Maybe they were your customers who no longer feel happy with paying you money. You are always only a part of the bigger picture, most of which you have no control over and only limited knowledge of.

Exactly. This guy replies later in the thread "there should never be a bug deployed to prod" essentially saying what does it matter

The idea that you can guarantee there's never any bugs in your code is incredibly naïve

I'd love to know what company he's worked at that never makes mistakes. Charity Majors who I think is very sharp says something similar about not being afraid to deploy on Fridays. I think she emphasizes more that your observability is on point, developers own their deploys, rollbacks are easy and deploys happen often. I do think those are all things to aspire to but it takes a good deal of effort to make it all happen.

Reply on news.ycombinator.com