What baffles me is how they got his far into operations with such dreadful practices, 100-200k could have got them a really solid CI pipeline with rollbacks, monitoring, testing etc,
But spend 200,000 on managing 460,000,000? No way!
But spend 200,000 on managing 460,000,000? No way!
In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production
"CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other.
It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.
"The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess.
Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything.
What I observe more often is something like this though:
1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag.
2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now.
3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B.
4. Turns out for use case A we need to do something more but things should stay the same for B.
5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B.
What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases.
edit: formatting
A hard lesson to learn, and a hard rule to push for with others who have not yet learned.
Imagine what our species could do if experience were directly and easily transferable...
Same goes for functions, classes, React components, DB tables and everything else.
Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top).
If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ.
My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.
Either way, it shows some attempt at longer-term thinking undone by short term implementation done improperly. That’s a microcosm of this story as a whole which gives this incident a fractal quality.
This is a little inevitable when working with (internal) binary protocols. You have some bits that used to be used as one thing, haven't been used for that thing in awhile, and it can be very tempting to just repurpose those old bits for new tricks.
In that case I'd call the sin deprecating and reusing in a single step. If you have to change the meaning of bits in a binary protocol, you should deprecate the old meaning, wait several release cycles, and then repurpose it after you're very, very confident that there are no remaining clients or servers in lurking around production with the old meaning, and that you won't need to roll back production to the old state ever again.
Roll-out failures like this happen all the time; you have to roll-out new changes almost assuming they will happen.
While something may be a rounding error to the company, you can't get someone at the correct political level where it's a rounding error to pay attention.