How to Accept Over-Engineering for What It Really Is

These motivations seem to be behind many of the almost constant calls for rewrites, but rewrites with no better justification than this rarely turn out to be a good use of resources. It is particularly galling to see a rewrite that is no better than the original.

Certain (usually early) stages of a project may be time or resource rich or permit wide-ranging changes with minimal risk (client impact, downtime, live failover requirements, carefully engineered change-overs, etc.). At these times, an experienced person might opt to utilize resources to 'over-engineer' for contemporary requirements in the knowledge that it is a strong and well-reasoned long term play, while to the less experienced, non-technical project managers or stakeholders this may seem questionable.

Ultimately, every project is different but there is a 'sliding window' of project resources (particularly time and the availability of low-risk incremental change) and requirements (expanding, shifting, contracting).

A famous quote from a famous figure on migration: Those change-over things are really severe. Really severe problems. - Joseph Henry Condon, Bell Labs.[0]

Perhaps ultimately, over-engineering may be inefficient but efficient if considered in light of the alternatives: by parable, eventually including a rewrite in javascript that also reads email.

[0] https://github.com/globalcitizen/taoup

Most over-engineered monstruosities I've seen in my career don't come from engineers trying to uber-optimize their solutions to a sick level, but from noob/unexperienced/dumb programmers trying to get something done.

Things like: methods with just one line of code (to call another method), classes that seem to be there just to satisfy some dubious OOP pattern, too many levels of error handling which can be simplified by just removing them and catch the exception in a single place of the program (at least in exception-based languages), too much 'public' elements in a class (justified by the programmer thinking: "just in case" someone needs to access them in the future), excessive interface use due to the (clueless)need of pedantic levels of unit testing, NIH-retards creating frameworks/ORMs/libraries for every single component their software needs (I even saw once a tech lead that replaced all uses of hashtables in the company by his own implementation which, hey, let you look for keys in a case-insensitive way!), choosing low-level languages to implement non-performance-critical software, and a big etcetera.

In my experience many software developers completely fail to challenge the 'why' of what they're being asked to do, and jump right to design/coding.

To be fair though, not all software developers work in an environment where they can, or feel they can, ask or challenge the 'why', but the software will only benefit if they do.

I had a computer in the garage on my workbench. So I decided to solve this problem by writing some code. About an hour and some number of shots later, bullshitting with my friend the whole time, I had a working scoreboard program.

We continued playing darts. We were both amused that even though I was too wasted to subtract, I could still write code.

We probably should have just switched to Cricket.

With experience and instinct, some people are successful more often than not in predicting what level of engineering is sufficient, and they go on to become great engineers and leaders. But I am not wholly sure if this is more skill or more luck, though I am sure there is some skill.

With software, it is especially hard. Especially as you go deeper and deeper into the components. At the UI level, it might seem easy to say what is the amount of flexibility you need. When designing a common utility or infrastructure piece to be used by everyone in a large company, it is much less obvious. But I think the hardest layer is the mid-tier, which has to deal with the backend and the frontend, and support all kinds of changes and pluggability.

This is so wrong, in the context of software at least.

Over-engineered software has layers of abstraction that add nothing to the function or form of the product the costumer sees - it is over-engineering for the sake of ideological notions about design patterns etc.

As for functionality delivered to the client, software very, very rarely gets over-engineered - in reality it almost always gets delivered late and with core features still in development. The idea that there is all this software being delivered on time that not only meets 100% of all the original requirements with exacting quality and perfection, but is somehow delivering in gross excess to the point of giving customers more than they need is pure fiction.

Following this principle code example 2 (i.e. Javascript after deduplication) would be the sweet spot. It fulfills the requirements without in the most obvious way, at a good level of abstraction (the requirements did not make any of the three pictures specific -- defining a separate function for each actually goes beyond the requirements, while obscuring the higher-level goal).

All of the subsequent examples focus on using the features of the platform rather than clearly showing the purpose of the code, i.e. unnecessarily obfuscate it.

Even worse, if your under engineered product is causing problems, you'll almost certainly notice it (rather quickly too). If you over engineer, you may never realize it.

In the beginning people are clueless and just try to make things change and take a horrible amount of time for it. Then they start to develop some patterns. Then they see patterns from other people and choose one as their one true hammer that will hammer all the nails. Then he spents an awful time arguing why X and Y are nails. Then he sees that not all methods apply all the time and starts to balance them with each other. And finally if he gets that far, he will start to develop his real own, efficient methods.

So where is the over-engineer in this? Not in the psychopath section but in the hammer-and-nail section. He developed far enough to be effecient in some ways but still waists quite some time on trying to make all the problems to apply to his solution. That's normal. That's also about the skill level that you SHOULD have when you leave a university type of education. In soccer that's the point where the usual children club stuff ends and they start to compete in serious competitions, maybe one age level below the one that gets public attention. You want to put people on this skill level against real tasks to realise that not everything is a nail.

So instead of always blaming people for being in this level we should appreciate it. Our university education seems to work for that kind of job quite well. And then we should continue to put these people in the right environment to develop further and provide value to their teams despite the things that they still lack.

I can think of languages like C++ that facilitate what I would use a more derogatory term for like autoerotica. The site's example, taking Javascript seriously, almost always falls into this category.

In something like an automobile, using boxed steel everywhere would make the thing sturdy as hell at the expense of fuel economy and less objective things like aesthetics. A bridge design might be too expensive, or a material too heavy to facilitate the architectural vision. Etc.

In software this term is used where the software is actually under-engineered, like a shoddy deployment without any consideration of consequences. Technology like a message queue or distributed database are the victims of poor understanding or oversold claim. Almost always the real problem is clueless management and a symptom of fad driven architecture.

So going back to the materials example.. What software wouldn't benefit from end to end back pressure? Circuit breaker fault management? Memory safety? Formal proof? None of these things should have any bearing on the user, and there are a lot of companies could afford to figure out how to make one or more of these things have minimal effect on front side of SDLC with very positive effects on the long tail. Software is almost to fluid and inexpensive to be over-engineered.

It's not necessarily a bad thing in a technological sense, but it's bad for business.

If people can build an extremely powerful, complex system which works well and is stable, then that's great.

Some engineers will stay away from any system which they regard as 'complex' or which doesn't strictly adhere to their technical philosophies about simplicity.

Personally, I think that if enough engineers invest their energy into building a highly complex system, I think eventually they can make it work.

You can just look at operating systems; they are extremely complex but so much human effort has been invested in them that they have become quite stable and have become essential to the functioning of our society.

This is how I feel about newer systems like Docker and Kubernetes - I think that together, they are extremely complex. When Docker started out, I thought it was crazy, I didn't see the utility value; but enough fools got involved that it eventually grew into something meaningful. I think that the offerings we have now with Docker, Kubernetes, Swarm, Mesos, etc... have become extremely compelling.

I think often it's regarded badly in engineering when you have a single component that does a lot of different things (often, that's what people often call over-engineering) - But look at the general-purpose CPU; we didn't need to invent it; people could have been satisfied with building specialized electronics for every kind of machine that just did one job well (no programming needed) but this would have slowed down innovation (because manufacturing circuit boards is expensive and time consuming).

Making generic components that do a lot of different things CAN be extremely useful when you look at the big, long-term picture. It just has to be done well and needs to be given the time to stabilize.

The part that is counter-intuitive, is that a rather simple idea can sometimes have very complex requirements.

It's the way many bridges having a posted weight limit of 10tons can handle a 12ton truck without incident.

If you go to all the trouble of actually engineering something to begin with, might as well include some effort into making it stand the test of time while performing beyond expectations. One of the most fundamental features of true engineering is a "margin of safety".

Traditionally this is what increases the chances of your product having that most useful feature; "worth buying".

Without these gambles, the project will often fail too.

It's a really rather difficult problem.

Granted, my experience is more with hardware than with software, but getting a design "just right" can be costly in analysis and testing, and there's a palpable concern that if you aim low rather than high, a substandard product will get pushed out the door. It can be exponentially more costly to correct a weak design late in a project than to over-design it early: Subsystems and schedules become more inter-dependent, the closer you get to launch.

I can give you a good design in six months, or a bad design in twelve. ;-)

1. Make sure engineers are engrossed in the business case. This sets proper scope.

2. Don't generalize something ahead of time, only after you do the same thing twice. The architecture should support easy refactoring, and there should be certain discipline about actually doing the refactoring the second time around.

There are many challenges to overcome with both of these, but it will be worth it.

Under-engineering is the problem for anything that's supposed to last.

You get hired by this startup with great ideas. They have really hit their niche nicely and are getting more and more customers. You build a great throwaway prototype and customers and the mighty CEO love it. You ask when you are going to rebuild the prototype, since it's a bit of a mess... The hipster CTO says that there is no need, that you are the greatest dev ever and that he will get a tattoo with your name.

The first change request comes in. Easy. It's still straight-forward to make some quick and dirty changes here and there. You call it a day and your hipster CTO throws a party to celebrate the amazing job.

After change request 10, you feel like slowing down drastically. The mental model you could fit in your brain until now, is too complex. You can't touch all the involved parts of your application anymore or it would cost you a lot of time. You get away with some if/else statements to fix the edge cases. You feel happy again, you did it! You call it a day, your hipster CTO pats your back and gives out some beers.

After change request 20, the system is a mess. Different layers of the application talk directly to each other, thus creating painful coupling. Your abstractions are either non-existent or all in the wrong places. You have no CI, because that would have been over-engineering the test suite. Your local tests take half an hour, because making them run in parallel would have been over-engineering. The time for documenting the project was spent hunting quirks somewhere in the darkest places of your application. You don't even try to touch some of the code you wrote just a couple of months ago. "There be dragons" is, next to "TODO", the most used comment you find in the codebase.

So after many days, the job isn't done yet. The mighty CEO gets nervous and starts running all over the place, giving you some "leadership". He says that there is a deadline not to be missed or the universe will collapse. The hipster CTO even has to skip his yoga-classes to meet the deadline. After many extra hours and even more energy drinks, you get the change request done. The hipster CTO just smiles at you, but no beer this time.

There is 10 new bugs but you don't know about them yet. Soon you will, because people will start complaining about that last change you did and the bugs you introduced. By this time, you are sick and tired of it all. You hate yourself. You feel guilty. Problem is, you don't even have the time to think about how tired you are. Request 30 is going to arrive soon...

But hey!, don't forget that you HAD TO ship it quick, not think about solving the problem! Abstractions? Over-engineering! Designing for resilience? Over-engineering! Small, independent services? Over-engineering! Staging system? Over-engineering! All these thoughts come to your mind everytime. Yet, you are so tired, you fall asleep and try to forget it all...

WAKE UP! THERE IS A BUG IN PRODUCTION!!!

You need to rush. In the middle of the night, your messy apartment gets illuminated by the blueish light of your laptop. You don't even care anymore.

The End

---

Appendix

By change request 40, your soul committed suicide.

By change request 50, the company was spending 80% of their time doing bug fixes and keeping their self-build house of cards running.

By change request 60, there was this tiny competitor who gathered together a nice little team of skilled and motivated engineers. The CTO would give them time to think about how to solve the problems best. Soon they entered the market at an amazing pace. All competition went out of business. You lost your job. You have no motivation. You feel lonely.

By change request 70, which never actually happened, that tiny competitor sees you as a valuable asset to their company. Since they are growing, they need people like you, with experience and skills. You still feel like the worst developer ever, but you take the job. People respect you. The software was designed following best practices. It is stable, people iterate quick. You can take your time to think about the best solution. People share their ideas and with each iteration they make it better.

Finally, after all, you are happy again...

In my experience (mostly in networks) these are two different things with a negative correlation between them.

One can't lump safety in with poor engineering.

seems to me the article is talking about something akin to "wrong engineering"

Anyway, among many of the big issues here are so many things are contextual - the problem domain, stakeholders, programmer abilities, business climate, etc. It's been my experience that you cannot apply judgement to a solution without understanding as many of the variables involved as possible. Shifting these variables even slightly can change everything.

For example, say I build a feature A at company Y and then I change jobs next week and to company Z which is much smaller. Company Z wants the same feature I just built at company X, but of course I can't steal it outright. Building the same thing at company Z requires different engineering decisions, given the different climate and probably different development team, programming language, and so on hypothetically. Even if I could just copy the solution, at company Y it might be perfect, while at company Z it might be considered over-engineered. It's a combination of both objective and subjective judgements.

As for the concrete implementations of features, I often am driven crazy by both over and under engineering. I worked with several people in positions of power above or horizontally to me that would routinely shoot down many well-engineered features and implementations by other people because they were, "Too complicated." I began to realize this was often shorthand in most cases for, "I know nothing about this, my ego or political stance is threatened, I lack talent, and/or I am simply a moron." Of course I also hit plenty of actually over-engineered things that were indeed exactly this, but in many of these cases it was obvious. Where it is less clear is anything of decent scope, challenge, subjectivity, or other hard to quantify measure. I think people can look at something like the FizzBuzz and see that the parody versions using OO 10 design patterns with abstract factories are obviously over-engineered, while it gets much harder in the context of a big project.

And this leads to another problem. Complexity is not over-engineering, nor vice-versa always. Nor is scope, or code size, or any of these related concepts that come to mind. A very small amount of code can be incredibly complex, a large amount of code can be simple, and so on. Either can be over or under engineered. Just look at math - some of the most useful tools are very difficult to understand, while others are incredibly simple with tons of layers, nuances, and so on. Programming is not that much different in this regard. We've all seen things done in a tiny amount of code we would have never thought of and maybe can't even understand. Likewise, we've encountered huge amount of code that someone complains about but is written clearly, thoughtfully, and cleanly. These's a good related talk on some of these ideas by Rich Hickey if I recall.

Another issue is that it is hard to quantify both knowing what and when justifies substantial engineering efforts. As I mentioned, context plays a big part, as does intuition and experience (in relation to ability, not just pure years). These are all nearly impossible to quantify, and yet must play a large role in cases where large engineering decision making. A lot of people see things they don't understand or are new to them and just throw out the over-engineered card. Likewise, a lot of people love pulling out their favorite tool (ex: SQL, Framework of choice) and try to use it as a golden hammer. Anything else is "over-engineered." These things can lead to over-engineering or at the very least, poor engineering.

Over-engineering is also something that is relative. It is really hard sometimes to work with someone else and to get them to understand especially those hard to quantify things such as intuition. Programming can also often highly be about ego and ownership. People will not hesitate to point at something not written by them as over-engineered because it was not done how they would do it.

Game programming for me again comes to mind as an area where this is a constant pain. The inexperienced game programmer will both under and over engineer things like you've never seen. First, they'll try to use their favorite language to write a game. They'll then do things when they learn a technique like OO programming and try to apply it everywhere, to everything. "A monster? Obviously I should make a monster base class, then subclass this for an Orc, then subclass it again for an Elite Orc." Everything must be designed around the tools they know instead of the problem. The minute this person does enough in their career to get any real authority or a real job, they take this unchecked mentality and try to impose it on the people around them. I see the same thing in web application and desktop programming. Someone learns lambdas, functors, currying, design patterns, IOC, monads, etc. and they must apply them to any situation that triggers this recognition. The "engineering" then comes from the solution and tools, perhaps the XY problem in many cases. Usually these people also fail to consider the aforementioned contextual concerns.

I know in game programming this happened often where some programmer would come in and replace some lengthy looking code that made heavy use of arrays and manual for loops. They'd then go in and replace it all with vectors, iterators, and all kinds of heavier stuff. They never considered that the code was engineered that way for performance or for portability or backward compatibility or some other reason. Moreover, they didn't actually know that the seemingly "simpler" solution was actually making things as a whole more difficult by pushing out these concerns to be fixed or handled elsewhere (ex: making up performance somewhere else).

On a personal note, I am often annoyed by the over-engineered finger pointing. There are so many things like XML messes and 200 steps to setup something for a simple purpose that are easy enough to agree on as over-engineered. Most of these come down to asking yourself, "What problem am I trying to solve" and perhaps repeating those steps aloud. If you feel ridiculous saying them, especially to another person, something might be wrong. But if it turns out that the problem you are trying to solve really does require something that is not a sound-bite, the problem explanation will also often have to be long and cannot be skirted around without ignoring the problem.

Too often I see things in this industry thrown out there to "fail hard," to be an MVP, to test the market, to disrupt, to fill a gap. That's all fine, but make sure you are solving the problem and not creating new ones like security, performance, misleading people, defrauding investors, or worse because you thought engineering a proper solution is too hard or takes too long. Find the sweet spot and make informed decisions. I recommend another talk about Rich Hickey (sorry two by the same person, just memorable talks) about Hammock-driven development and thinking before acting. If more people did that, it could swing either way and maybe we would have less short-term "stuff" but better long-term solutions, less over-engineered messes, and things that actually are worth building on top of as bases. As it stands now, this industry is littered with poorly engineered solutions we were stuck with because of both over and under engineering.

The project in the comic actually appears to be building a tree swing. I'm sure this was just an oversight on the author's part but if anything it reinforces the point the comic is making.