Seriously, were there shortcomings in the software design of yesteryear? Almost certainly. Are they blown vastly out of proportion in most discussions? My assertion is that they are.
Sadly, I do not know as that I have anything coherent to offer on how to fix things. What I can offer is that I think we'd do better by not always looking across the horizon to a silver bullet language/framework/whatever and keep focused on what the job at hand requires. The amount of failures I have witnessed due to a desire to over generalize is staggering.
Software design had a lot of cool things happening in the 1980's (possibly more in the 1970's, though), but the mindset has shifted since then. How many people are implementing their own VM's? How many are running on non x86 intel/amd chipsets? Things that needed to be heavily optimized in the early 80's might actually run faster now with JIT & whole-program optimizations. And so on.
It's hard to build an engineering discipline out of such a shifting substrate.
I don't know what to say regarding your examples. Don't put too much credit in JIT and whole-program optimizations. One could just as easily point to the fact that what used to be too slow of a program is now fast enough with the advance of computing speed. And memory. Do not overlook the importance of memory.
More, I doubt most programs of yesteryear failed due to lack of optimization. Seems likely to me, that the main cause of software failure has not changed that heavily in the years.
I find that they (from the '50s and even older) hold up just fine.
If software worked initially, and survived 60 years that well as bridges from the 50s do today, that would be a miracle.
For the first question, I would only be offering speculation. Google and friends can give pretty good references.
For the second, a quick google gives "The annual direct cost of corrosion for highway bridges is estimated to be $6.43 billion to $10.15 billion ..."[1]
Third, I offer TeX as a good example of old software that has managed to survive for quite a long time. I am always pleasantly surprised when I go to typeset something from 20+ years ago and things "just work."
I think the theme here is that maintenance costs for bridges more often than not entails just keeping them working. Seems that far too often in software maintenance costs try and include complete rewrites into new technologies.
[1] http://www.dnvusa.com/Binaries/highway_tcm153-378806.pdf
Onto bridges, just because they build them differently now does not negate the fact that 50 year old bridges are still reliable and safe to drive over because they were engineered well. Their shortcomings and failure modes are well known so maintenance can be performed to prevent collapses.
50 year old bridges may be expensive to maintain at this point, but the fact that it's still possible shows that they were engineered well. They may build them differently today, but that's more likely related to cost constraints changing (e.g. can't afford an army of riviters now) rather than the engineering being unsound.
TeX is not 50, I had not meant to imply it was. Just that it is a relatively old piece of software that has had virtually no maintenance compared to many other pieces of software.
Could it have used an overhaul? It is certainly debatable. However, Knuth has had massive success in keeping it working without worrying about use cases that just were not the aim of software.
To continue the comparison with bridges, it is not uncommon for them to have restrictions saying that large trucks can not cross them. This is not a recommendation, it is a requirement. In the software world, many would have modified TeX to do such things as typeset documents miles wide. Because, generality! Or some such.
And the concerns you have about people not knowing what an "underfilled hbox" means is simply a lack of training. I would be surprised if anyone I work with below the age of 25 has even heard of TeX, much less read the documentation for it. Heck, I could probably raise that to 40.
So, my assertion is that the fact that TeX is still very much usable shows that it was "well engineered." Would it require somewhat expensive training? Sure, but how is that any different than the maintenance of bridges?
Given a Fortran compiler I see no reason why lot of the numerical stuff in netlib would not work just as well in 60 years time: http://www.netlib.org
So, then should we assume that software is like road building? No, rather I would assert that there are some pieces of the road that are more isolated from dangerous use and changes than others. In software, we have some of that, too, but not to nearly the extent. We are often holding up pitchforks to rewrite everything.
It might actually be a good allegory after all...
Remember, we've been building bridges for thousands of years. We've had a lot of time to figure out how best to do it. The amount of improvement we've had in 50 years in terms of technique is probably pretty small; improvement in materials is the more likely reason for any variations in technique.
I don't get to deal with LISP anywhere enough to substantiate these claims from actual experience, but that is the impression I have gotten from hanging around places like this.
If anything, I think the dream from back then would be that embedded systems of today would be more FPGA based. Screw this using fixed design systems. :)
As for the dream of dropping fixed systems, you can already buy a cheap fast mcu from xmos with 4 cores which you can use 3 to create extra peripherals.Or another cheap CPU with a small fpga. But of course if you're fighting for pennies of cost or power, there might be better fixed alternatives than those 2.
BTW the reason we don't have FPGA on chips is probably mostly commercial - charging by the peripheral offer chip companies ways to more revenues and more differentiation between each other. They really don't want to sell the same commodity chips of mcu+fpga.
I don't accept that generics are somehow an automatic win for languages. I do like them somewhat, myself, but I don't think there has been compelling evidence that they can be used to great effect in building embedded systems.
Consider, at the low level, the majority of the code written is still in C. Not C++, C. And I can't bring myself to agree that it is lacking because of it.
I know he is intentionally inflammatory, but I think Torvald's rant on why he rejected C++ for Git is somewhat poignant. As is all of the attempts at writing the git core in higher level languages. Specifically, how they haven't really succeeded.
I will confess to being open to the argument, but outside of revisionist history and wishful thinking, I just don't see proof that things are automatically better with today's practices over those of yesterday's.
Second, I don't think anyone would argue that the biggest thing spurring movement in GCC is the success of clang.
The other data point regarding rust is the huge excitement in the embedded community.
And do not mistake what I'm saying as that I think rust isn't as good as c. Even better, in many ways. My question is specifically if people today will be able to accomplish better things because of the language. I'm doubtful.
Most of the biggest accomplishments in embedded space comes down to the massive gains in the silicon. Most of which is dominated by advances in boolean chain evaluation to make faster circuits. Wider, sure, but also faster. (At least, that is my understanding... I'll admit I am no authority on this.)