Bad scientific code beats code following "best practices" (2014)

Different scientific research fields are using widely different computer software environment, and have their own habits and traditions. The way a biologist uses programming has no reason to be similar to the way an astrophysicist does: they have not at all experienced the same software environment. It may even be useless to talk about "scientist" in the same field as two different labs working in the same field may have very different approaches (but it's more difficult if there are shared framework).

So, I'm not at all surprised that you observe opposite experience. The same way I'm not surprised to see someone saying they had the opposite experience if someone says "European people are using a lot of 'g' and 'k' in their words" just because they observed what happened in Germany.

I see a lot of opinion/taste presented as something more, but I really can't think of superstitions.

For example, Snakemake (os-independent make) with data version control based off of torrent (removing complication of having to pay for AWS, etc) for the caching of build steps, etc would be a HUGE win in the field. *No one has done it yet* (some have danced around the idea), but if done well and correctly, it could reduce the amount of code and pain in reproducing work by thousands of lines of code in some projects.

It's important for the default of a data version control to be either ipfs or torrent, because it's prohibitive to make everyone set up all these accounts and pay these storage companies to run some package. Ipfs, torrent, or some other centralized solution is the only real solution.

I think a single semester of learning the basics of software development best practices would save a lot of time and effort in the long term if it was included in physics/maths university courses.

It is easy to learn some coding, not so easy to become a scientist.

To becomes a scientist you must write and get your PhD-thesis approved, which must already be about scientific discoveries you have made while doing that thesis. Only people with above average IQ can accomplish something like that, I think.

Ultimately I’d say the core issue here is that research is complex and those environments are often resource strapped relative to other environments. As such this idea of “getting shit done” takes priority over everything. To some degree it’s not that much different than startup business environments that favor shipping features over writing maintainable and well (or even partially) documented code.

The difference in research that many fail to grasp is that the code is often as ephemeral as the specific exploratory path of research it’s tied to. Sometimes software in research is more general purpose but more often it’s tightly coupled to a new idea deep seated in some theory in some fashion. Just as exploration paths into the unknown are rapidly explored and often discarded, much of the work around them is as well, including software.

When you combine that understanding with an already resource strapped environment, it shouldn’t be surprising at all that much work done around the science, be it some physical apparatus or something virtual like code is duct taped together and barely functional. To some degree that’s by design, it’s choosing where you focus your limited resources which is to explore and test and idea.

Software very rarely is the end goal, just like in business. The exception with business is that if the software is viewed as a long term asset more time is spent trying to reduce long term costs. In research and science if something is very successful and becomes mature enough that it’s expected to remain around for awhile, more mature code bases often emerge. Even then there’s not a lot of money out there to create that stuff, but it does happen, but only after it’s proven to be worth the time investment.

That's not on them though. That's on the state of the tooling in the industry.

Most of the time, dependencies could just be a folder you delete, and that's that (node_modules isn't very far from that). Instead it's a nightmare - and not for any good reason, except historical baggage.

The biologists writing scientific programs don't want "shared libraries" and other such BS. But the tooling often doesn't give them the option.

And the higher level abstractions like conda and pip and poetry and whatever, are just patches on top of a broken low level model.

None of those should be needed for isolated environments, only for dependency installation and update. Isolated environments should just come for free based on lower level implementation.

For someone to shake these bad practices, they need to fight an uphill battle and ultimately sacrifice their research time so that others will have an easier time understanding and using their codes. Another battle that people trying to write "good" code would need to fight is that a lot of academics aren't interested in programming and see coding as simply as means to an end to solve a specific problem.

Also, another bad practice few bad practices to add to the list:

* Not writing documentation.

* Copying, cutting, pasting and commenting out lines of code in lieu of version control.

* Not understanding the programming language their using and spending time solving problems that the language has a built in solution for.

This is at least based on my own experience as a PhD student in numerical methods working with Engineers, Physicists, Biologists and Mathematicians.

This leads to a split between domain problem solvers, who are driven to solve the field's actual problems at all costs (including unreliable code that produces false results) and software engineers, who keep things tidy but are too risk-averse to attempt any real problems.

I encourage folks with interests in both software and an area of application to look at what Hadley Wickham did for tabular data analysis and think about what it would look like to do that for your field.

As ever, the best work comes when you're able to have a tight collaboration between a domain expert and a maintainability-minded person. This requires humility from both: the expert must see that writing good software is valuable and not an afterthought, and the developer must appreciate that the expert knows more about what's relevant or important than them.

You can't solve the first 3 issues without having people who care about software quality. People not caring about the quality of the software is what caused those initial 3 problems in the first place.

Firstly, a lot of the programs people were writing were messy, but didn't need to last longer than their current research project. They didn't necessarily need to be maintained long-term, and therefore the mess was often a reasonable trade-off for speed.

Secondly, almost none of the software people had any experience writing code in any industry outside of research. Many of them were quite good programmers, and there were a lot of "hacker" types who would fiddle with stuff in their spare time, but in terms of actual engineering, they had almost no experience. There were a lot of people who were just reciting the best practice rules they'd learned from blog posts, without really having the experience to know where the advice was coming from, or how best to apply it.

The result was often too much focus on easy-to-fix, visible, but ultimately low-impact changes, and a lot of difficulty in looking at the bigger picture issues.

I'm not sure what "uninstallable" code is, but why does it matter? Do scientists really need to know about dependencies when they need the same 3 libraries over and over? Pandas, numpy, Apache arrow, maybe OpenCV. Install them and keep them updated. Maybe let the IT guys worry about dependencies if it needs more complexity than that.

> Writing completely unreadable code, even to themselves, making it impossible to maintain. This means they always restart from zero, and projects grow into folders of a hundred individual scripts with no order, depending on files that no longer exists

This is actually kind of a benefit. Instead of following sunk cost and trying to address tech debt on years-old code, you can just toss a 200-liner script out of the window along with its tech debt, presumably because the research it was written for is already complete.

> Foregoing any kind of testing or quality control, making real and nasty bugs rampant.

Scientific code only needs to transform data. If it's written in a way that does that (e.g. uses the right function calls and returns a sensible data array) then it succeeded in its goal.

> They are also less productive when coding than the scientists because they care too much about the quality of their work and not enough about getting shit done.

Sooo...another argument in favor of the way scientists write code then? Isn't "getting shit done" kind of the point?

* "Big ball of mud" design [0]: No thought given to how the software should be architected or what the entities that comprise the design space of the problem are and how they fit together. The symptoms of this lack of thinking are obvious: multi-thousand-line swiss-army-knife functions, blocks of code repeated in dozens of places with minor variations, and a total lack of composability of any components. This kind of software design (or lack of design, really) ends up causing a serious hit to productivity because it's often useless outside of the narrow problem it was written to solve and because it's exceedingly hard to maintain or add new features to.

* Lack of tests: some of this is that the scientist-turned-programmer doesn't want to "waste time" writing tests, but more often it's that they don't know _how_ to write good tests. Or they have designed the code in such a way (see above) that it's really hard to test. In any case--unsurprisingly--their code tends to be buggy.

* Lack of familiarity with common data structures and algorithms: this often results in overly-complicated brute-force solutions to problems being used when they needn't have and in sub-par performance.

This quote from the author stood out to me:

> I claim to have repented, mostly. I try rather hard to keep things boringly simple.

...because it's really odd to me. Writing code that is as simple as it can be is precisely what good programmers do! But in order to get to the simplest possible solution to a non-trivial problem you need to think hard about the design of the code and ensure that the abstractions you implement are the right ones for the problem space. Following the "unix philosophy" of building small, simple components that each do one thing well but are highly composable is undoubtedly the more "boringly simple" approach in terms of the final result, but it's a harder to do (in the sense that it may take more though and more experience) than diving into the problem without thinking and cranking out a big ball of mud. Similarly reaching for the correct data structure or algorithm often results in a massively simpler solution to your problem, but you have to know about it or be willing to research the problem a bit to find it.

The author did at least try to support his thesis with examples of "bad things software engineers do", but a lot of them seem like things that--in almost every organization I've worked at in the last ten years--would definitely be looked down on/would not pass code review. Or are things ("A forest of near-identical names along the lines of DriverController, ControllerManager, DriverManager, ManagerController, controlDriver") that are narrowly tailored to a specific language at a specific window in time.

> they care too much about the quality of their work and not enough about getting shit done.

I think the appearance of "I'm just getting shit done" is often a superficial one, because it doesn't factor in the real costs: other scientists and engineers can't use their solutions because they're not designed in a way that makes them work in any other setting than the narrow one they were solving for. Or other scientists and engineers have trouble using the person's solutions because they are hard to understand and badly-documented. Or other scientists and engineers spend time going back and fixing the person's solutions later because they are buggy or slow. The mindset of "let's just get shit done and crank this out as fast as we can" might be fine in a research setting where, once you've solved the problem, you can abandon it and move on to the next thing. But in a commercial setting (i.e. at a company that builds and maintains software critical for the organization to function) this mindset often starts to impose greater and greater maintenance costs over time.

[0] https://en.wikipedia.org/wiki/Anti-pattern#Big_ball_of_mud

    >    * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months

Ultimately, if there were a simple way to get data in the correct state in an os-independent, machine independent (from raspberry pi to HPC the code should always work), concise, and idempotent way - people would use it. There isn't. But the certainly could be.

The solution we desperately need is a basically a pull request to a simple build tool (make, Snakemake, just, task, etc) that makes this idempotent and os-independent setup simple. Snakemake works on windows and Unix, so that's a decent start.

One big point is matching data outputs to source code and input state. *Allowing ipfs or torrent backends to Snakemake can solve this problem.*

The idea would be to simply wrap `input/output: "/my/file/here"` in `ipfs()`, wherein this would silently check if the file is locally cached to return, but if not go to IPFS as a secondary location to check for the file, then if the file isn't at either place, calculate it with the run command specified in Snakemake. It's useful to have this type of decentralized cache, because it's extremely common to run commands that may take several months on a supercomputer that give files that may only be a few MBs (exchange correlation functional) or only a few GBs (NN weights) so downloading the file is *immensely* cheaper to do than re-running the code - and the output is specified by the input source code (hence git commit hash maps to data hash).

The reason IPFS or torrent is the answer here is for several reasons: 1) The data location is specied by the hash of the content - which can be used to make a hash map of git commit hashes of source code state that map to data outputs (the code uniquely specifies the data in almost all cases, and input data can be included for the very rare cases it doesn't) 2) The availability and speed of download scales with popularity. Right now, were at the mercy of centralized storage systems, wherein the download rate can be however low they want it to be. However, LLM NN weights on IPFS can be downloaded very fast when millions of people *and* many centralized storage providers have the file hosted. 3) The data is far more robust to disappearing. Almost all scientific data output links point to nothing (MAG, sra/geomdb - the examples are endless). This is for many reasons such as academics moving and the storage location no longer being funded, accounts being moved, or simply the don't have enough storage space for emails on their personal Google drive and they delete the database files from their research. However, these are often downloaded many times by others in the field - the data exists somewhere - so it just needs to be accessible by decentralizing the data and allowing the community to download the file from the entire community which has it.

One of the important aspects to include in this buildtool would be to ensure that, every time someone downloads a certain file (specified by the git commit hash - data hash map) or uploads a file after computing it, they host the file as well. This way the community grows automatically by having a very low resource and extremely secure IPFS daemon host all of the important data files for different projects.

Having this all achieved by the addition of just 6 characters in a Snakemake file might actually solve this problem for the scientific / data science community, as it would be the standard and hard to mess up.

The next issue to solve would be popularize a standard way to get a package to work on all available cores/gpu/resources, etc on a raspberry pi to HPC without any changes or special considerations. Pyspark almost does this, but there's still more config than desirable for the community, and the requirement of installing OS-level dependencies (Java stuff) to work on python can often halt it's use completely (if the package using pyspark is a dependency of a dependency of a dependency, wet lab biologists [the real target users] *will not* figure out how to fix that problem if it doesn't "just work"[TM])

Here you are not improving your time to get out an article, but reducing it for others - which will make your work more influential.

Here's is where I disagree. It's detrimental in the short term, but to ensure reproducibility and development speed in the future you need to follow best practices. Good science requires good engineering practices.

I never understood nor understand people who nest their inner loops in an entangled mess of hardly distinguishable digits, which is error prone.

Same for method names.

I try to use speaking out loud to some of my methods: What do you do? And if the answer is getValue I believe it needs renaming.

My intent is not to put down maintainers of scientific software! It's super cool and super important.

I see the damage a person decades in an industry can do when they cluelessly and energetically start to test and implement a new shiny thing on an industrial codebase.

When the product brings in hundreds of millions a year, there is incentive to patch up the damage so you can have future releases and continue the business. I'm not sure how much resources a scientific codebase maintenance could use just to patch up a mountain of architectural and runtime damage.

It appears that you are not even talking about the same problem as the author. You seem to be talking about people who all define themselves as programmers, some of whom have more experience than others. The author wasn’t talking about new-hire programmers, they were talking about experienced physicists, chemists, biologists, etc., who have been doing some programming, possibly for a long time.

Either way, most of my experience is with all-programmer teams, and I have to say I’ve seen the experienced programmers make far bigger and costlier messes. The people who can’t really program might always make a lot of messes, but they make very small messes, and nobody puts them in charge of teams or lets them do that much process critical work without oversight or someone re-writing it. I’ve watched very good very experienced programmers make enormous mistakes such as engaging in system-wide rewrites that turn everything into a mess, and that cost many millions of dollars, only to take years longer than they estimated, and to come out the other end admitting it was a mistake. There was also the time a senior programmer tried to get really clever with his matrix copy constructor, and caused an intermittent crash bug only in release builds that triggered team-wide overtime right before a deadline. He was incredulous at first when we started to suspect his code, and I had to write a small ad-hoc debugger just to catch it. I calculated the dollar cost of his one line of cleverness in the several tens of thousands of dollars.

Anyone who in 2023 still thinks inheritance is a good idea for anything other than a few very specialised use-cases is not somebody who seriously cares about the craft of software development, not somebody who's put any effort to study programming theory and move beyond destructive 1990s enterprise Java practices. Widespread usage of inheritance inevitably makes code harder to reason about and refactor, as anyone who's compared code in Java to code for similar functionality in Rust or Go would see (both Rust and Go deliberately eschew support for inheritance due to the nightmares it can cause).

How maintainable code is is measured in how well you know where to change something, and how certain you are that it did the right thing without side effects.

The fatal error of the linked article is that bad scientific code often suffers from correctness problems - not just theoretical concerns, but the "negates the main point of this paper" kind of thing.

Recently at work some people argued "things" (methods, classes, even files) should have a limit in size. I think that's valid thinking because you want to strive to having smaller components that you can reuse and compose, if you are able to do that. But what happened is that people started creating dozens of little files containing a function each and then importing those. To be it's obvious that this is now a lot worse because the complexity is still the same, just spread out across dozens of files. But most people were somehow convinced that they're "refactoring" and that they're doing best practices of keeping things small.

Who are these people, really? I mean, I was that for a rather brief period of like a year, somewhere 3-4 years into programming IIRC. I met others like it. But in the end people seem to learn and grow out of it, and rather quickly so, because it becomes clear what the issues are quickly. Really good learning by mistake though, wouldn't have wanted to miss out on it.

mantra

That's imo just another mistake to make: wrapping a rather strict and narrowly scoped principle in a paradigm-like-must-be-followed mantra hurts the right-tool-for-the-job idea, which feels vastly superior.

Firstly the idea that unmaintainable code is necessarily an issue is already wrong to start with, in my book. Obviously it's not ideal and where appropriate - meaning nearly always - should be avoided at pretty much all cost, but I have enough examples where it does not matter at all. As in: code which hasn't been touched in 20 years and probably won't ever be touched. Does it look like a nightmare? Yes. Does it work correctly? Yes. Does it need changing? No. So, is it an issue where spending time (or having spent time) on it would make anything other than the programmer's peace of mind (well, or ego perhaps) better? Clear no.

Secondly: of course I get where you're going with such definitions but it again lacks the very much needed nuancing. I can write unmaintainable code in one file but where you need to change 20 different locations. You could than claim that your mantra still applies because the code should have been split over 20 files, but yeah, that's what you get with mantras :) Likewise depending on the feature it's perfectly possible many files have to be changed but that doesn't necessariy mean the code is hard to maintain. Could try to claim that it wasn't very well architected to start with, maybe, but welcome in the real world where not everything can be thought of from the very start except in small toy applications.

Trivially false: if all of the source code is kept in one file then you only ever have to edit one file. No matter how many lines of code are in that file, any such system would be maximally maintainable by your definition.

Maintainability is not so clear-cut. It's currently an imprecise measure of organizational clarity that makes it straightforward to extend a system in ways that are needed or will be needed.

Wrote fancy math on a PhD economist’s laptop.

Then when we added testing when developing a platform, we found out we’d been 5% off in allocating Prime revenue between organizations — and had the correct amount gone to Retail, the 2018 hiring freeze might have been avoided. (According to a very angry Wilke.)

Whoops.

Turned out we did need a team and all those guardrails, processes, code reviews, etc.

There’s a time and a place for “move fast and break things” — but real trouble can come from taking those academic practices into the real world.

Not sure how much of a problem it is, but I am frustrated when "other" engineers try to teach me about how networking works and never once consider that my intuition may possibly have more value than theirs, because I actually studied networking. Not that I am always right of course, but if we have an electrical engineering argument, I naturally get into a stance where I assume they know better and can teach me useful stuff.

and on top of it those DriverControllers and ManagerControllers keep getting depracated

But perhaps the most relevant response is that few people read papers, even fewer are going to look into their zip. The whole idea of papers is to condense a whole lot of work into concise digestible information.

Write some code and generate immediate results can be considered a done job. If you want to reuse the code a year later and found an unreadable mess though, not so much.

Sure, you can ignore security if all you're doing is processing local text files, granted. But things looking horrible to programmers isn't just about security bugs, it's about the whole span of correctness bugs. And scientists need to write code that is both correct and maintainable. The frequency with which they don't is partly why results so often can't be replicated, making the money spent on academia wasted.

The idea that science code doesn't need to be maintainable or that they have some magic way to do it that looks wrong, isn't right either. It's not uncommon to find model "codes" that scientists have been hacking on for decades. The results have become completely untrustworthy many years earlier, but they deny/obfuscate/ignore, attack or even sue people who point out concrete problems. Sadly, often with the acquiescence of the media who are supposed to be ferreting out coverups.

Scientists need to collectively get a grip on this situation. They will happily attack anyone outside their institutions as being non-expert conspiracy theorists, but when it comes to software they suddenly know everything and don't need to hire professionals. Paper-invalidating bugs are constantly being covered up and the only reason the problem hasn't reached criticality yet is that many people don't want to hear about it. But the unreliability of academic output is now becoming a political problem and a divisive culture war issue, when it really shouldn't be. A good first step to solving the replication crisis would be for scientists to stop pretending it's OK to quickly knock together a program themselves instead of assigning a ticket to a trained full time SWE. Yes it would cost more (a lot more), and that's OK. Generate fewer papers but get them right!

I think the goal is to make if more accessible, so that more people with less knowledge can produce more crap with it.

People don't learn the basics, they want to write a comment in Copilot and have it assemble code that roughly does what they want. I believe that people who got into software in the 60s actually liked computers. People who get into software today just want to produce stuff, they don't care about their computer.

As for hiring software devs, that's not going to change (in general, there are places where software devs write code used by scientists, but rarely are these codes themselves pushing research boundaries, it's code on top it that does) absent significant changes in funding structure and rules (which are typically a government/public service concern, and not up to researchers).

Most scientific code is horrible from any sane software developer perspective. The quality is so bad that I think a huge proportion of published results are plain wrong due to bugs in the analysis. These apply to my much of my code as well.

But a lot of "software engineering" code is horrible too. Mostly because most popular technologies and "best practices" are just plain bad. Overenginering is a pandemic and has been a long time. Much of the roots is from the gilded age of enterprise Java. Totally misunderstood OOP. Byzantine layers of pointless abstraction. Rampant premature "web scale". Counterproductive bondage and discipline (yes, including much of static typing).

These have become a cargo cult in software development, and these cosmetic features are deemed as "quality code".

And this leaks into scientific programming. Even Python, arguably the language of science nowadays, and a "quick and dirty one", forces some of this cargo cult. Modules are needlessly complicated (e.g. relative and absolute imports are quite a mess), let alone the horrible packaging system. And the current trend to push typing.

That said, scientific code is getting slowly better, largely due to switch to public/open source code, and away from specialized hacks like MATLAB and R. Especially in more technical fields.

In software engineering OTOH things are IMHO getting worse.

That's not the distinction. Good software engineers solve problems. That's what the paycheck is for. The distinction is whether code has to be maintained.

It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper's been published.

It's _often_ the software engineer's job to build things that deliver business value over years, evolving and expanding requirements, in a development team, without grinding to a halt under the weight of accumulated complexity.

"Quality" software engineering is just heuristics for keeping the pace of change high over time without breaking things.

But that 'clear boundary' thing is a naive bollocks! No such thing!

Both domain experts need to understand things beyond this imaginary and when precisely drawn then highly arbitrary boundary that is more like a gradient than a line normally (also not something relevant in a final good product).

Teams consisting members unwilling to wander into foreign territory and expecting to be fed and deliver over a strict boundary will do horrible things!

(we are not even trained this way btw., professions have quite a bit of overlap and we learn matters others are expected to take care of in practice, and in case of sofware engineering - apart from the most notorious ignorants no one wants to work with - it is pratically impossible avoiding to learn a big chunk of a foreign profession on the fly for delivering a good product, heavy science is no exception!)

It’s our job to deliver value to the business at a rapid and maintainable way. Rapid changes are often worth more to the business than maintainability, even over a period of many years. In some cases you could put a non software engineer down and let them build something with ChatGPT and it would work perfectly fine for the next 5-10 years because it’s focused on something particular, doesn’t change much and lives in isolation from the greater IT landscape. In other cases all your points are extremely valid.

That being said we also work in an industry where a lot of “best practices” often turn out to be anti-patterns over a period of a decade. OOP is good in theory, and I know why we still teach it in academia, but in practice it very often lead to giant messes of complexity that nobody really knows how works. Not because the theory is wrong, but because people write code on Thursday afternoons after a week of no sleep and a day of terrible meetings.

After a few decades in the industry, what I personally prefer isn’t any particular approach. No, what I prefer is that things are build in isolated services, so that they only handle their particular business related responsibilities. This way, you can always alter things in isolation and focus on improving code that needs it. It also means that some things can be build terrible, and be just fine, because the quality of the tiny service doesn’t really get to “matter” over its life time.

I personally write code that follows most of our industry’s common best practices. Because it’s frankly faster once you get used to it, but I’ve seen really shitty spaghetti code perform its function perfectly and never need alteration in the 5-10 years it needed to live before being replaced.

The problem is not that "physicists write code". Scientists end up learning a lot of stuff and getting good at it (software engineers the same). The problem is that writing software is often left as a job for the occasional phd, postdoc or research assistant, ie people with temporary positions, and in general to people who see building software as a side duty at best, annoyance at worst. This results in no generational knowledge building, being hard to find mentors, less learning and reflecting on practices on how to build software, rediscovering the wheel constantly, too much effort put into . It is not that one graduates as "software engineer" or as a "scientist" and then they know the best practices and everything of their respective fields. People get to learn stuff. Software engineering practices should be in the culture of scientific software building, not simply carrying them from the software engineering world to science, but adapting them and taking it each own idiosyncrasies as a field.

  1. code for performing a scientific simulation or analysis (a script).
  2. code for solving a specific problem generally (a program).

Sounds like the mission of Research Software Engineers (https://society-rse.org/).

I work as a software engineer (with a PhD in a field that is not CS) in a research setting, and there is a give and take. 50% of my job is reading, understanding, and adapting very bad code from non-software engineers into a production system. But another 50% is binning the overwrought inflexible code written by my software engineer predecessors in order to do all that more quickly than refactoring would allow.

In a research setting, in my opinion, MVP is king. Researchers seem to usually not produce viable long-term solutions. But software engineers do as well by virtue of not being domain experts (how could they be, they would need a PhD to understand the research domain!) and being unable to test 100% of the assumptions underlying the software themselves. Which is why it can help to have someone in between who is a domain expert, but knows just enough software engineering to produce production-good-enough code.

Precisely! See my relevant comment from another thread here - https://news.ycombinator.com/item?id=38821679

References:

1) Why science needs more research software engineers - https://www.nature.com/articles/d41586-022-01516-2

2) Research software engineering - https://en.wikipedia.org/wiki/Research_software_engineering

Look at machining for example. Long dominated by people working in machine shops making tools and parts on the clock. But in the background there’s a strong hobbyist contingent and there you’ll find endless debates over whether a beginner should buy a decades old Bridgeport milling machine or a brand new Chinese-made one.

Similarly I find endless debates about what sort of frying pan to use (nonstick, stainless, carbon steel, or cast iron) among amateur chefs, but you’ll never see people working in a restaurant waste time on that.

In other words, it’s hobbyists who really obsess over tools and craftsmanship. They do it because they love it. Programmers just happen to be among the odd sort who can get paid to practice their hobby.

I think that most software engineers' jobs is not to build quality software, it is to make money. When you are trying to make money, the goal is not necessarily to make the best quality software that you can. Often, it is to make acceptably good software as soon as possible. A company that writes software that is half as good and ships it twice as fast might outcompete a company that writes software that is twice as good and ships it half as fast. I sometimes wish that my job was to build the best quality software that I can, but that is not the case. What I really get paid for is to make my employers' company successful, to make them money. And it's not that my employers don't care about making high quality software, it is just that if they cared about it too much they might get outcompeted by others who care less about it.

That is not to say the kind of software engineer that does what you say and tends to build quality software (or at least move it in that direction) doesn’t exist, but the demand for people who can make computers do stuff (loosely, developers) so far outstrips the supply that outside of a few bubbles (HN being one of them), they appear to be so vanishingly rare they might as well not exist.

I’m sure some scientists get to work with useful and highly valuable professional developers, but I’d be amazed if they were the majority.