Why Is There So Much Crap Software in the World
software.rajivprab.com
software.rajivprab.com
Obviously there is also bad free software out there, but it has more benign afflictions. Free software also tends to steadily improve with time, but commercial software is a roll of the dice with every update.
Is it a smart business move? Debatable (I'd argue overdelivering on a SaaS product also helps bring in new customers). From a moral standpoint, I find it incredibly difficult to justify penny pinching and filling hours with malpractices and bureaucracy.
And for what it is worth: personal experience tells, a lot of customers are quite accepting of faulty software, as long as the damage can be reverted and minimized. Even those in finance. Wouldn't be the first time I see people casually solve a bug that caused millions worth of damage.
The difference is just one of speed and scale. The software starts paying for maintenance almost immediately. People see a open source codebase that is now "good enough" for their problem, and within months it becomes a popular dependency, and then a singularity. Construction that goes wrong, on the other hand, often has a narrative that plays out over decades. The featureset of a bridge or building grows little with time, and often slims down, and so while the maintenance can be physically costly, it isn't like the building gets remodelled and rewired each year to do a new thing, which is a thing that happens often to software.
Software companies that can really embrace the "maintenance of a singularity" role usually last, even if they accrue some tremendous technical debt: see Microsoft, Oracle, Adobe for some examples. Companies that try to elude that outcome, on the other hand, have to have a business model of disposability: Apple tends to obsolete software when new hardware hits, as do most game companies. The companies that let the product rot are the common failure case, and the companies that keep the product small are themselves small.
When I see the current hurricane of ever-new and ever changing (evolving?!) software, the need to use an endless variety of differing products and technologies (including many libraries one has no control over), and endless discussions about the subtle differences and advantages and disadvantages of these competing / cooperating technologies.... How can anyone become really expert? How many can even become the fabled "full-stack" developer??
To me this really does not seem fair to the developer, especially those in smaller shops, where the developer almost seems to have to perform as a "full-stack" developer, whether they are or not. Just my 2 cents - I hope my impressions are wrong, but the app dev world sure seems scary!
Splitting the level of developers in three equal groups makes it look like it will take the same amount of time to get from noob to mediocre as it takes to get from mediocre to good. If I were to draw these lines the first one would be after a few weeks. You don't need more time to write useful software. You maybe can't solve hard engineering problems after that time, but it's enough to work in a team and be productive.
I'm not sure where to draw the second line, though. I don't know if there's a point where you simply stop making stupid problems. You get better at solving problems and might shoot yourself in the foot less often, but to write bug-free software, you really need dedicated QA who will concentrate on that. I don't think it's a problem that can be solved by individuals, but only in a team.
And this leads me to the conclusion that software quality is mostly driven by the company and economics. If the project doesn't pay for solid quality assurance or has too much time pressure on developers, the software you get might be bad at least to some degree. Even good developers can't make up for that.
I'm not sure this makes any sense outside my current bubble, so please add your 5 cents.
I think this is absolutely the case. As an extreme example, an organization that uses formal methods is probably going to ship higher quality software because they've meticulously worked out design bugs before they've even written a line of code and, once the program is implemented, they've rigorously verified its correctness – the economics of this methodology, however, are not really feasible in most circumstances and so instead we opt for duct tape fixes and crunch time.
The author also mentions the lack of gatekeeping for software engineering – this is often pointed to as one reason for bad software existing. I tend to agree, since "good code" and "bad code" that both do what they're supposed to under the common case can appear the same to the user and the manager who signs off on it (excluding bad UI). Point being, if someone can code just well enough to get by in their field, they will probably remain employed. This is a tricky problem to solve though, because so much of the practice of software engineering is qualitative (bordering on aesthetic, I would argue) and involves designing abstractions. While much of software engineering is analytical and must obey a certain set of axioms, higher-level systems and software design is really more akin to art and architecture than civil or mechanical engineering.
The reason there are so many terrible software developers is simple: the world tolerates the software they write.
This feature isn't limited to software. As Theodore Sturgeon put it, "90 percent of everything is crap".
Software is (generally) much cheaper than a bridge too. And a lot of buyers are especially aggressive in cost-cutting.
Having a proper profession for Software Engineers doesn't sound so bad.
The author torpedoes his entire thesis with his footnote:
> I am certainly not suggesting that we gatekeep the software profession
More like, enforcing certain standards. ie "When you hire from X, you can assume they know Y and are proficient with Z". It already happens with recruiting. When I see resumes from X I know what to expect.
If someone decides to engage in a race to the bottom and hires "engineers" from something other than X then, good luck. Maybe you found a diamond in the rough, but likely not. In that case don't whine that it's impossible to get nice software.
1. Bloated and useless functionalities. More functionalities will make any software more complex. If these functionalities are not well thought or lack logic, it's more difficult to implement.
2. Believes versus empiric proof. The big difference we have with other engineering field: we deal with believes. The building of a plane rely heavily on the laws of physics, we rely on the law of SOLID. The first is a scientific discipline, the second is a guy talking on a stage.
3. Ego-centered field. We all think we know better. We all think we have the truth ("Microservices are better!", "OOP is the best!") even if we have no idea about it.
This article is full of believes without any study to back it up, with a cool looking curve which doesn't measure anything. The hero expert programmer who will save us all is a myth. We all struggle, we all make mistake, and we build things together. The group synergy is more important (see rework from Google).
If you are building a crud system for a small business they are unlikely to pay enough to make it work spending time to make the interface easy to use. The admin staff are cheap, developers are not.
Small Business people don't want to spend money on testing and continuous integration for apps when they don't have large amounts of capital around. Nor do they push for security audits when they see no immediate value.
They want the cheapest piece of software that will get the job done. Not until they feel the pain of cheap development will they start to think about reworking their software.
Yes there are some that can produce incredible software in a short amount of time but these are exceptional people. Those people should be demanding a higher rate.
Most hiring I've seen in Australia has a lot of resistance to scaling remuneration to output.
Most software is underspecified (an unsolved problem) and thus in exchange for "hacking it together somehow" the vendor is allowed to disclaim any responsibility.
1. A developers job is more complex than most, and takes a very long time to learn, because you have to become proficient in so many aspects of the job. e.g.
- technical stack which is huge (languages, protocols, how hardware works, networking etc),
- plus technical tooling (ides, source control, platforms such as clouds, programming frameworks and so on and on)
- programming models (desktop ui, web ui, state management, apis consumption, api creation, security models, and so on)
- teamwork (project management systems, soft skills, team dynamics, professional writing and so on)
- user level software (operating systems, office software, communication software, anything users use that you need to interact with)
- understanding your industry (how all the above changes and evolves over time)
- one or more business domains - to know what software to write and the nuances of how to create it to be truly useful
- all the other areas, such as high performance, or high quality, or design for testability, or how systems fit into companies management frameworks, or designing a system for manageability over the span of it's lifetime, or for easy extensibility and so on and so on
2. Skill levels
- This is a hard ask, many people don't have the kind of mind to ever do this well, and will be mediocre forever
- Learning time, it tends to take 5-10 years for talented people to become proficient in ALL the above areas they need to combine to truly be a journeyman developer
- True mastery requires a commitment to learn to a deep level on so many area, and 10-20 years is probably realistic for smart people with the right mindset, to truly get to a level of masters of so many related disciplines
3. Skills triangle
- The industry has been growing so fast. There are ten or a hundred times more recently started developers than there are developers with decades of experience because as an industry software development is growing by orders of magnitude as every company becomes an IT company where in the past it was rare.
- People often move up and out. Of the rare developers with the capacity to become real masters, who then have the 10-20 years of experience to have time to learn it all deeply - many of them have left the field by that time. This is for various reasons such as: having made their fortune and moving to investing, getting sick of computing or the industry and wanting a change, moving across to management, not updating their skills to keep up with technology etc
- Quick napkin guesses to put metrics to these -- 1/10 developers with more than 10 years serious development experience (90% less than 10 years, because of fast growing industry) -- 1/3 developers with capacity to really master it -- 1/3 of those remaining in development instead of moving up and out
= on these completely made up but semi realistically guessed figures, 1/10 * 1/3 * 1/3 calculates out to having 90 non master developers for every master developer in the industry
So a true master developer is very, very, very rare.
~
As well as being very rare, they are other factors which mean their experience and guidance is not always utilised:
- management that often suffers from lack of experience technically
- peverse commercial incentives as one of the other comments here mentioned
- fashion storms of technology that occur due to the hype of a significant proportion of the hundred times as many less experienced developers filling the blogsphere and management mindshare with hype regarding the latest thing they've re-discovered that is already known if they cared to try to learn more deeply instead of try to 'thought lead'
- ageism and overemphasis on the 'new shiny' vs tried and true techniques
Last points might seem a bit bitter but please stay on topic, they are mentioned because they are significant, real reasons why there is so much crap software in the world!