Non-political crazy things uncovered in Twitter whistleblower report | Hacker News Reader

Non-political crazy things uncovered in Twitter whistleblower report

twitter.com

46 pointssavanaly14 comments

> Twitter does not have separate development, test, staging, and production environments.

It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it...

It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.

I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks.

I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.

With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful.

What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod.

Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes.

I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.

I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation.

At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.

There is zero chance this is true though. None.

You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.

> Disengaged CEO: CEO Jack Dorsey had recruited Mudge personally. They got along well, and Mudge has never suspected Dorsey of harboring bad intent. But Dorsey, the high-profile CEO of one of the most prominent companies on earth, was experiencing a drastic loss of focus in 2021. Dorsey attended meetings sporadically, and when he did, he was extremely disengaged.” In some meetings —even after he was briefed on complex corporate issues — Dorsey did not speak a word. Mudge heard from his colleagues that Dorsey would remain silent for days or weeks. Worried about Dorsey's health, the senior team mostly tried to cover up for him,” but even mid- and lower-level staff could tell that the ship was rudderless

Worried about Dorsey's health, the senior team mostly tried to cover up for him

This seems like a failure of the corporate model in general. 'Everyone knows' the CEO/dictator/monarch is unable to function, but senior leaders cover it up (poorly), the board looks the other way, shareholders are deceived or at least kept ignorance, and the rabble (that's you) are confused at best and dismissed as crazy if they intuit the state of affairs correctly.

Why do we institutionalize such systems when the patterns of failure are so glaringly obvious and have been endlessly repeated?

How parag isn’t being prosecuted boggles my mind.

It's amazing to me that you haven't been prosecuted.

Edit: Some people really failing to see the point here, which is that it's meaningless to suggest someone should be prosecuted without specifying why.

Edit: Wrong thread. My bad

Wrong Thread?

Yep wrong thread thanks! Mind throwing me a downvote? I can't delete due to reply.

Flagging is the best option in this situation. I flagged your original post (and upvoted this one). That eventually makes it disappear along with the replies if there is not much traffic.

Did not know that. I don't have the karma to flag comments it seems.

Reply on news.ycombinator.com