But also, I think there has to be some conceptual basis for us to be able to think about factoring in a clear way, and that kind of intellectual labor is hard on another level.
Eric Evans' book on domain-driven design has some examples of programming tasks that generate confusing and unclear code, until the coder together with the "domain expert" reaches some kind of epiphany—like "ohh, if we think of these three things as being parts of the same kind of thing, then this part of the program makes more sense."
A program is a kind of philosophical theory about a certain topic, and if the theory isn't very clear, then you'll end up with a lot of special cases and obscureness even if you have unit tests and refactoring tools. They say naming is the hardest thing in programming and that becomes more true when you start to introduce any level of abstraction.
"Design patterns" were an attempt to create more general small theoretical pieces, above the syntactic level. In the FP world, a similar vision is that algebraic and categorical abstractions can provide some of those pieces. We still haven't seen much of what algebraic abstractions can do for ordinary programming...
But these abstract pieces, I think, are pretty crucial if we want programs to be understandable as anything but arbitrary collections of procedures and types. So yes to refactoring tools, and yes to more discussion about design patterns—it didn't end with the Gang of Four! read Christopher Alexander and Richard P. Gabriel!—and yes to more inspiration from mathematics and engineering and philosophy and all kinds of stuff, to provide inspiration for factoring—yes to study of logic and language and ... I'm getting carried away.
1. https://en.wikipedia.org/wiki/Lisp_%28programming_language%2...
Only if those refactoring tools could refactor the domain itself (and the environment), not just the code :-)
Often these special cases aren't intrinsic to the code, but caused by inconsistent business rules, or weird issues with the environment (some browsers not sending some events, workarounds for compiler bugs on ancient platforms that must still be supported, socket closing might or might not cause a flush depending on whether you're on Windows or *nix, ...)
Getting to a deeper understanding of what the old code is doing takes a lot of work. When we get to that point, refactoring to make what we learned apparent is a good idea. Assuming there is a way to test the changes. (Now I get to spend 2 days making "mocks" in environment X -- learning the mockup tool, and determining the content appropriate for the use cases)
It's a good point you bring up, but it's not a freebie. I am in favor of doing what you said, when you can.
Most systems I've worked on have some overriding architecture and assumptions. The worst quality erosions happen when changes are shoehorned into the code without a complete understanding of those factors: it's likely there'll be semantic mismatch between the new feature and existing components, unexpected bugs, and harder to modify code (because the overall architecture is no longer coherent).
You can mitigate the disintegration by having a maintainer with a strong understanding of the architecture and the problem(s) being solved by the original system. Talking with them can give you a roadmap on how to modify it properly, and if it's even possible to do so while maintaining architectural and design coherence.
Note: my comment is an opinion based on personal experience. I'm not trying to play it off as fact.
Somewhat the same arguments apply to approaches which put a lot of emphasis on small-scale/unit testing -- they offer reassurance when doing small-scale refactoring at the expense of making substantive refactoring harder.