Three ways to solve hard programming problems
jvns.ca
jvns.ca
Every time I see this, I find that the problem of leaky abstractions result in a 3 phase solution. It's awesome in the beginning, everything "just [mostly] works". Then a few months in, you start to feel like a young dutch male with his finger stuck in the earth, holding back the ocean. In the end, you throw it out, having "stolen" the important bits for yourself.
Sometimes, you find yourself stuck at one of the first two phases. I wish it was more frequent.
It just feels better to build such a solution as an engineer. Stories about these solutions are much more interesting than "I used a ready box and saved the company two weeks of my time so I can spend it on another feature of a prototype". And usually, the better technical level of a software developer, the more he prefers the long, right, rigid way to do Things That Last.
But from entirely business perspective, this is actually good advice in a surprisingly big share of the situations.
Given that, it makes sense to, in that environment, default to "let's do this thing correctly". But that's not actually long, right, and rigid. It's often shorter than one might think, and far more flexible, as it usually means "let's do the minimum amount of features that we need, but what ones we do, let's make sure we do well, working with the user to define them, and get feedback/iterate, writing lots of test, documentation, etc along the way". The problem is that businesses tend to prioritize exactly the opposite; give us features, now now now, that have only been defined at a very high level and not actually thought about by anyone, and any time not spent on implementing features is wasted. And with that mentality, using an existing solution seems to make sense. It's only afterwards that you realize that solution brings a lot of unnecessary complexity, doesn't give you the level of control you want, didn't instill the necessary domain knowledge in the developers for proper maintenance and modifications etc, possibly didn't even solve the problem(s), but by then it's too late, you're just told to make it work.
For example, I often take the approach that my first implementation, which likely will end up in production, is used for learning the interface, service or business need. It satisfies the problem as it currently exists and was defined, and generally has a few hacky parts as the needs were explained in more detail or changed halfway or more through the project. The next time anything more than a bug fix or trivial feature is required, I'll rewrite the most problematic behavior in a more extensible and cleaner way, since I now have a much better understanding of the problem. This is generally quick, as I can re-purpose portions of the original that are perfectly usable, and I have a reference implementation (the original) to compare against to find any bugs in the new version.
I can rely on this because the chances that I will be around at this company in the future are extremely high. This would not be a good approach for a company that sees domain expertise leave, or move to different departments that aren't easily approached for information. I've been on projects where you come in to rewrite on someone else's old code because nobody really understands it well, with little or no information besides the code itself and included comments as to the original design decisions. It's easy to identify the portions of the code that look like they were stupid decisions, and you would definitely never do that. You start writing code, and soon you run into unexpected conditions that require you to make the same or equivalent hacky choices, and you understand a little of the original developer's prior choices. You'll end up with the code rewritten, and possibly a little better and cleaner, but if someone comes along in a few years, they'll have the same problems you did. These are the situations you want to avoid by spending more time up front to make the solution better in the beginning, or at the bare minimum, documenting in the code the hacky abstractions you've been forced to make.
First time a senior dev overrided my choice, and to my dismay we did a pure DB solution that took longer and was far inferior in its version control capabilities than if we had done it in VCS/XML.
Second time I was in charge and pushed this "great idea" through. We ended up with awesome branching, cherry-picking, diffing, etc capabilities (which were not in the requirements and nobody really used), but ran into big problems later when we needed more thorough data consistency checks, incremental non-GUID id's, speedier centralized inserts/updates, etc. Second step was rube goldberg, and third step was rewrite given the actual requirements using a normal db store (probably half the code, better performance, easier to use, no "extra" VCS functionality that nobody needed).
Lesson learned: don't use VCS as a data store unless it's kicking you in the face. Or, maybe there's a market for a VCS with some DB-like properties.
In fairness, I think a more intellectually honest critique would have been to write the script using literate programming. Sure it might have required McIlroy to write a tool for literate shell programming -- the will to do so being the magic trick separating Knuth from mortals. The world might have been better for it.