Code Janitor: nobody's dream, everyone's job

I think it's hard to write beautiful code straight off -- it very rarely happens to me, anyway. So for me, writing beautiful code is mostly perfective maintenance anyway -- or do I misunderstand?

   It's obvious that making changes to the set-up in the second pictures will take less time than modifying the first picture's set-up.

I don't think your conclusion is as concrete as you make it out to be. One's knowledge of OTP likely correlates with general programming ability and experience. Better, more-experienced programmers probably find it easier to learn new code bases, so they don't need as much ramp-up time from management.

Overall though, good post. Kudos for gathering data!

I find it mind-blowing that multi-million dollar companies who are struggling with growth cannot see the quality of their software engineering actually prevents growth. They try rewrites but with the same approach to software engineering as always, it is doomed to fail.

Software is like a garden. It can be master planned, well manicured, organized or it can be the result of uncontrolled organic growth of years and years.

Either way, my job is to keep it healthy and servings its purpose. It is my job to water,fertilize, pull out weeds as they show up and sometimes to uproot whole trees and plant them somewhere else as they are necessary for the system.

Continuing with the analogy, it is key to any good gardener to have good tools to do the job, because they make all the difference on how hard you have to work and how much you can get done in a season.

If you believe in self-documenting code, I feel like you need to do things like this because it's part of keeping your documentation up to date. (In our code base, we maintain separate documentation, but reading the code is essential to getting the total picture.)

Apologies in advance for what's going to seem like self-promotion. Many of my opinions I've already plopped onto the internet in my blog so I'm going to be doing a lot of blog linking.

Maintenance has a nasty paradox. To do it right, you need political power. You need to be able to say, "I am fixing this and that's that and any problems it causes for you are your problem." The level of political clout you need to do maintenance decently is manager-level. You need help from a lot of people and if you don't have the ability to force them to prioritizing helping you in this gnarly task, they probably won't. (Feature freeze? I need a Bob-ruling before I do that.) In most companies, maintenance is not rewarded, and it's not fun, so people with manager-level clout generally avoid such work. (http://michaelochurch.wordpress.com/2011/09/23/taxation-with...) The politically enabled people tend to take the "shiny new" work (and since no one has the clout to oppose them on anything material, they often made bad decisions and produce shoddy results) and the underclasses end up maintaining the garbage after they launch, get promoted, and flee the system. So maintenance ends up being a workball that gets dropped on people who don't have the power to do it right, don't have the expertise to know how to do it, and don't have any good reason to care.

People underestimate complexity density and how much it can vary, and also how high it can be in source code. How long does it take to read "a book"? Well, it's between 2 and 400 hours for me, depending on the material. 10,000 lines of code is a book. Sometimes it's boilerplate that you can scan, but more often, it's closer to the 400-hour end of that extreme.

IDEs don't solve this problem very well, because they tend toward "four-wheel drive" syndrome: they get you stuck in more inaccessible places. I think read-only IDEs are indispensable, but if I were running a company, I'd be tempted to disallow IDEs for new development. One of the perks of maintenance is that you get to use an IDE. If you're on new development, then as long as the code is good I don't care and you can use whatever tools you want, but if I see you writing obvious "IDE code" I am taking that shit away.

One thing that I think is at the root of the problem is a denial about what programming really should be. People say "math is hard", but math is actually easy. At least, it's an only-somewhat-hard way to do things that would otherwise be impossible. Imagine what it would be like to build a physics engine if calculus and linear algebra didn't exist. How would you generalize linear regression to N dimensions without matrix calculus? Quite often, the mathematical abstraction is the simplest (and most terse) way of representing and solving a class of problems. It's hard, because of the intrinsic complexity. Mathematicians are also familiar with the experience of taking a whole day to read and understand a 4-page paper. They know, from painful experience, how much complexity density can exist in 100 innocent-looking lines of code.

Why the FactoryFactory crowd exists is that there's a class of programmers who really don't belong in the industry because they despise or can't hack math (and another class of programmers who have the ability but have been misled into thinking that FactoryFactory bullshit is actually the right way to do things). So instead of mathematical abstractions, they use pointless business-y shit made up to turn the tables against smart people with taste. ("What? You've never used a MetaSelectorVisitorSingletonFactoryFactory? And you call yourself a programmer?") Of course, even though the individual pieces that commodity programmers create are less difficult to understand than appropriate mathematical abstractions, the overall tangle that must be generated in order to have a chance at solving the problem becomes incomprehensible quick. But (channeling a commodity programmer) "no one reads code" anyway. What, you expected insight into the problem in that shit? We just wrote that because our bosses told us to do it.

At any rate, your insight into the sociological problems associated with code maintenance is right-on. The work is given to the people who are least equipped to do it. So can't companies just allocate maintenance work to the best programmers? The answer is no. First, if the best programmers do maintenance work in a company that doesn't value that stuff, they're seen as just screwing around doing junior-level work. Second, no one good will tolerate being "allocated" to maintenance. Good people will happily maintain code they care about, but if you tell a good programmer that she has to maintain someone else's legacy or find a new job, she'll take the latter. So you need a culture in which good engineers choose maintenance and have the political power to do it right. It's really hard to make that happen.

Google is better than the vast majority of companies in this regard, but they still have a culture where it's well understood that promotions usually come from launches. Maintenance is relatively well-respected (by industry standards) at Google, but new invention is still better for your career. You can get promoted on maintenance (which makes Google better than most places) but your odds still aren't as good, and there's more downside to maintenance, where the worst-case outcome is macroscopic non-accomplishment (i.e. you beat your head against a wall and have nothing to show for it). There are large bonuses (sometimes 6- or 7-figures) for people who take on important legacy rescues, which is a step in the right direction-- an acknowledgment that it costs major dough to get anyone good to maintain someone else's mess-- but those usually go to managers and tech leads who oversee the rescue efforts; the engineers are "just doing their job" and usually try to transfer to something with more upside as soon as they can.

So the second point about developing a decent maintenance culture is that you need open allocation (http://michaelochurch.wordpress.com/2012/09/03/tech-companie...). You can't "allocate" good engineers to maintenance. They have to choose it, which means you need an environment where people are encouraged to work on what's important to them and to the company. I've taken to using a 2-by-2 matrix to explain this. First category work is interesting, important stuff: core machine learning algorithms, search at Google. Allocating this work is no problem. Second category is the important but undesired work. Third category is interesting stuff that hasn't become important yet: R&D-type projects best suited for "20% time". Fourth category is unimportant and undesirable work. Important legacy rescues are 2nd-category work. Under open allocation, companies have to create genuine incentives (not just "do it or I'll fire you") for people to do 2nd-category work, and 4th-category work doesn't get done (management isn't willing to pay or promote for it, because it's not actually important). But 4th category work shouldn't be getting done in the first place. It pisses people off and adds minimal business value.

By the way, closed-allocation shops generate huge amounts of 4th-category work (often the majority of the workload of a closed-allocation shop is 4th-category). If you have a closed-allocation shop, the 2nd and 4th category work gets glommed together in "that ball of work we (management) allocate to people we dislike" and not done well. This is a mistake; 2nd-category work actually is important (by definition) and shouldn't get the 4th-category treatment. It needs to be done by people who have the ability and actually care.

Third subrant, and then I'm done (for now). I don't know what OTP is, but these insights about modularity and small-program methodology are right-on. The Unix philosophy works. The only case I've seen of a program departing from the small-program philosophy and adding value is in the database world. Databases have very stringent requirements related to performance, transactional integrity, durability, and concurrency, and all the pieces have to work together. So, they become large systems that macroscopically function as huge, featureful programs. It has also taken some of the best minds in computer science decades to get this stuff right. Databases are hard, yo. Do you have the best minds in CS on your business-logic codeball? Do you have decades? If no, then avoid big-program methodologies outright.

If you use the big-program methodology, your programs end up becoming parochial because their surface area is huge and requirements get barfed on it and the code becomes a pile of political injections, not real software. You end up with Java Shop politics (http://michaelochurch.wordpress.com/2012/04/13/java-shop-pol...). However, to be fair, this problem has little to do with Java itself and is more accurately described as a problem afflicting large-program philosophies in general.