While idea to construct everything from components overall seem to work better than to build monoliths quite often (Space X vs NASA, Linux vs many other OSes) but not always (e.g. SQLite).
You can and you must do your best, but you can't do everything - you must choose what is your sacrifice.
Verified* instances in safe Idris. Of course GIGO is still valid and failures in lower-level components (such as a processor just not behaving as specified) can cause failures in higher-level components, but it's simply impossible for there to be a failure in that component itself - any observed failure will necessarily be the result of a lower-level failure.
More likely, you're not unaware of these possibilities, you're just using a meaning of "failure" that excludes them. But regardless of whether you call these failures or call them something else, they prevent you from guaranteeing perfection.
And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
Not my experience; the system may need new components or need to rearrange the existing ones, but the components themselves don't become imperfect. Consider something like a prime factorization algorithm; maybe your system will change to the point where you don't need it, or need to extend it, but it continues to be perfect.
> And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
Tech companies go bankrupt all the time, so I think it's a stretch to say that commercial coding is working perfectly. Automating a sloppy human process in an equally sloppy way can certainly make you money, and maybe there's more money in doing that than in making perfect things, but we should at least acknowledge that there's an unexplored alternative here.
> Tech companies go bankrupt all the time
generally speaking they don't go bankrupt because of bad tech, though. It's usually imperfectly understood customer needs. No amount of perfect tech is going to stop that.
Currently yes; that's what I'm arguing could change. Every library has to be written somewhere, and at a lot of jobs I've found myself rewriting the same should-be-standard pieces again and again.
> The stuff that's developed in-house is to service a customer need, and customer needs definitely change over time.
Yes and no. There are definitely parts of the system that change over time, but a lot of what gets written in-house ends up being surprisingly basic/generic things like database mapping, job scheduling etc.. You'd be surprised how often there just isn't a publicly available implementation of the thing you need even when it really "should" be part of the standard library.
> generally speaking they don't go bankrupt because of bad tech, though. It's usually imperfectly understood customer needs. No amount of perfect tech is going to stop that.
I'm not convinced. The way bad tech generally manifests itself is that the company is slow to react to changing customer needs, because it takes too long to make changes to their systems. Having been on the inside, I've definitely seen things that you would think from the outside are business failures but had poor technical choices at the root of why they happened.
What does it even mean for an internal specification to be "wrong"? The specification for the system can be wrong, and maybe the component wouldn't be useful in a correctly-specified system, but even then I think it's misleading to say the component is "broken".
Yes, such as the part that judges whether something is fallible or not.
If anything, every research on complexity and correctness shows that it does not compose and does not survive in a dynamic system. I would love to see the research that support this claim. It would be a revolution in these fields.
I would like to broaden it a bit and find out if there possibly is some truth to it.
The bad news: it depends a lot of what you mean, it may not be the answer you search and i do not have a "perfect" reference, because it is considered mostly obvious in these fields...
But before i get there, a few things.
> (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality)
Nearly all the references i point to will tell you that this leakiness and this constant changing reality is exactly what makes them work in the real world.
Now onto references. A lot of them can be found at http://resiliencepapers.club/
I particularly advise to start at https://www.youtube.com/watch?v=Pb_zYs8G6Co
https://www.youtube.com/watch?v=PGLYEDpNu60
For more in depth, i particularly like https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf
And this is basically a whole meta study of all the research and ideas that came to this field since WW2. Beware it is long https://www.sintef.no/globalassets/upload/teknologi_og_samfu...
For an example of how things go wrong when things are correct but interact well together, i advise these two papers on space systems. (I link to Adrian blog on them because they may be easier to read, but there is a link to the papers in there)
https://blog.acolyer.org/2017/11/30/the-role-of-software-in-... https://blog.acolyer.org/2017/12/01/analyzing-software-requi...
If these things make you curious for more, there is a lot. Or if you want to talk about/have help navigating what is in the http://resiliencepapers.club/ link, feel free to reach to me on twitter or by email. I will be happy to talk more.