There's practically an entire genre of "here's what I learned from my expensive mistake" stories. Usually they teach humility and caution. This one is interesting because it's a variant of YAGNI. Imagine that you've carefully tuned your (software/car) for a specific set of parameters, then throw one out of whack by mistake - does this get caught by monitoring? Do you even notice? If you hillclimb from the new status to an optimal one, do you get back to the same local optimum?
Then of course there's the famous RISKS digest, which can be funny or depressing depending on your mood and dependence on complex systems.
(My own personal one of these? "killall" behaves differently on Solaris than it does on Linux ..)
(It's not uncommon to use this tool to reconfigure some configuration values at runtime)
At least not on Solaris 10 or any of the open source derivatives. Seems it was special cased at some point in Solaris 11.
Had to make that mistake 2-3 times before I finally realised what I was doing to crash it given the slight delay.
Tiny example, GNU ls:
ls -lh bla : ok
ls bla -lh : ok
Most non-GNU lses, second form : oh-oh! :)
During one test run, one input did not update, we were happily feeding stale input to algorithm, which in turn took control paths that could have caused severe damages in unsupervised environment. Lessons learned: if you rely on invariants about inputs in code (update interval in this case), guard against violations of those invariants. This helped me many times later.
Every time you use assert() somewhere you're doing just that.
I guess rather well-known real world example is systems starting to show weird behaviour when database resultset ordering changes (e.g. changes in indexing) and someone relies on correct entry being first/last.
Assumed invariant here is resultset ordering and code is written with this particular invariant in mind. If not enough ordering clauses are used, resultset ordering is not invariant, even though it worked for years already :)
What that gains us is a run with real data and a build that's instrumented with assertions - which flushes out a lot of previously unknown issues.
So yes, you learn useful things. If you ever did it accidentally, the useful thing you learn is that shipping ideally is not an on/off switch, but a carefully monitored process :)
I'm working on desktop software, and this happens to builds that are full production builds, but that don't go to stable. So a very small subset of our users - who has agreed to being guinea pigs - occasionally gets these.
It also helps that we have a clear policy that assertions can only fire in cases that truly would be a bug in the wild - you can't assert and handle the issue. Either you think it might happen, then you handle it - or you state an invariant, in which case there's no reason to handle it being false. In other words, the only price you pay is that your build is slower - any assertion that fires would've been a silent bug before anyways. And, to be as close to non-debug as possible, any assertion firing merely creates a crash dump, it doesn't actually crash the executable.
And finally, we simply need to do it in the wild because many of our bugs are simply due to the staggering number of environments desktop software encounters.
We left some database healthchecks running on every request for a while, making load times a factor of 10 longer than they needed to be. Learnt our stuff was really fast and we didn't need to worry about performance much, I guess.