Comment by SavageBeast | Hacker News Reader

Comment by SavageBeast | Hacker News Reader

I'm currently refactoring a fairly large piece of research code myself. It was written with lean startup thinking in that a little code ought to produce some value in its results. If i was able to eeek some usefulness out of this code, then Id put more energy into it. Otherwise I was perfectly happy to Fail Fast and Fail Cheap.

How did it become such a mess in the first place? Simple - I didn't know my requirements when I started writing it. I built it to do one thing. In running it I learned more things (this is good - why you build stuff like this in the first place). The code changed rapidly to accommodate these lessons.

It wasn't long before I was running into limitations in the design of the underlying libs I was using etc. Of course I could find a way to make it work but it wasn't going to win any Software Design Awards.

Im happy to report that despite ending up a tangled mess, it actually helped me to come to understand and conquer a very specific kind of problem. In doing so I learned the limitations of commercially available tooling, the limitations of commercially available data, not to mention a great deal about the problem domain itself.

This research software has earned its keep and is now being cleaned up into a more organized, near commercial quality kind of project. Im glad I threw out "architecture" when I first started with this. It could have gone the other way where I had a very well built piece of code that didn't in fact perform any useful function.

I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable.

The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up.

Because changing your code could always break it, refactoring also reinforces the habit of writing code that can be readily tested -- also a good thing.

Makes you wonder why most languages don't come with good refactoring tools.

Safe automatic refactoring requires the ability to do static analysis of the code. Many refactorings are harder in loosely-typed languages.

Refactoring tools were invented in Smalltalk and worked just fine.

This has always surprised me, since I learned it.

What are the features of Smalltalk that allowed this to happen? Conversely, what is stopping this from existing in more modern dynamic languages?

Smalltalk has simple and strong reflective features. Moreover, it does not make a difference between the developed program and IDE. This means that doing things like that are very natural and well established in the Smalltalk cultural background.

Indeed. Having the whole system in front of you, and knowing how patterns like MVC or Thing-Model-View-Editor encapsulate parts of it, makes it very easy to "reason about" wholesale changes to the system.

Smalltalk is dynamically typed, but I wouldn't call it loosely typed. It's closer to Ruby's "duck typing". I was thinking more of Javascript and PHP. Weak and loose.

At least in the languages I work primarily in (JS and Java), I find my IDE to be pretty good at analyzing a lot of it.

Refactoring is kind of subjective, because there is rarely One Right Way to solve a problem, and you need context, so I could see why it’s not something that languages themselves take strong opinions on.

I don't think refactoring tools are that useful for refactoring, most of the time you are doing non-obvious refactoring tools can't help with anyways. Depends what we call "refactoring", tools are mostly useful for what I would call "housekeeping".

Well, every refactoring can be seen as a series of correctness-preserving housekeeping operations.

I should have added that I am probably abusing the term refactoring if it has a precise definition. What I'm talking about is working on improving the readability of my code, but also improving its structure. Today, "spaghetti" probably doesn't refer to a tangled mess of code sequences because we've gotten rid of the GOTO, but tangled interactions between modules, many of which are vestigial.

A lot of my code interacts with hardware configurations that will cease to exist when a project is done, but I mainly look at the stuff that's potentially reusable, and making it worth re-using.

I'm using Python, and there are a lot of tools for enforcing coding styles and flagging potential errors. I try to remove all of the red and yellow before closing any program file. I don't trust myself with too much automation! "Walk before you run."

That was my exact thought reading this.

I used to write some crazy spaghetti code as an untrained student working in a lab. Coding would go really quickly at first, but as I kept adding on to accommodate new requirements it became a huge kludgy mess.

Recently (after quite a few years of software engineering experience) I helped a researcher friend to build some software. He was following along with my commits and asked why I kept changing the organization and naming of the code, pulling things out into classes, deleting stuff that he thought might be needed later, etc. He spends only a small part of his time writing code, so he's never realized how much time it actually saves to keep things organized and well-factored.

I like Brooks' "plan to throw one away; you will, anyhow.":

This [first] system acts as a "pilot plan" that reveals techniques that will subsequently cause a complete redesign of the system.

However, in practice I'm not confident enough in my understanding, and fear losing all that hard-won work, so I refactor too.

A rewrite from scratch is probably more viable when the project is small enough to keep in your head at once.

Good architecture is pretty much just about slicing things up so that rewrites / refactors can happen incrementally, rather than all at once. This can actually go both bottom-up (these functions are easy to re-arrange and don't need a rewrite) and top-down (these functions suck, but I don't have to rearrange anything to replace them).

"Good architecture is the one that allows you to change."

What if you slice it up wrong?

Then you're gonna have a shit time of it, and may need to do a total rewrite, instead of an incremental one.

Brooks has since amended this[1] to say that he really meant it in the context of traditional "waterfall" development, where the first iteration is meticulously planned and designed as a whole system before any code is written at all.

Rapid, iterative prototyping, followed by refactoring, is a perfectly reasonable approach today. No need to create a fresh repository and rewrite all code from scratch.

David Heinemeier Hanssen, creator of Rails and a big advocate of building working code as early as possible, wasn't even born in 1975 when the mythical man-month was written. Linus Torvalds was a (presumably) plucky 6-year-old. Brooks wrote that book for an audience that would have known waterfall as the only way.

[1] https://wiki.c2.com/?PlanToThrowOneAway

wikiwiki seems to be having trouble moving to github. Here's wayback https://web.archive.org/web/20190821162524/https://wiki.c2.c... And the bit you referenced:

  "This I now perceived to be wrong, not because it is too radical, but because it is too simplistic. The biggest mistake in the 'Build one to throw away' concept is that it implicitly assumes the classical sequential or waterfall model of software construction."

You don't have to throw everything away. Like reusing libraries, you can reuse your modules, or whatever code you like.

It's the part that is a tangled Gordian Knot that is easier to cut than meticulously unravel.

"Starting with a clean slate" is a common idea in many contexts. One very similar to code is writing. You can hack and edit, but a fresh draft is easier (and safer) for fundamental conceptual and structural changes.

BTW "Waterfall" was a parody of processes, though the conceptual aspects (requirements vs specifications etc) are useful. People then were just as intelligent as today. Maybe more.

I don't see waterfall as a product of limited intelligence but rather limited tools and history. It's natural that early software engineers would generalize processes from mechanical and chemical engineering, especially given the cost of iteration with early software tools.

We are talking about research code here. And from many of the comments, it seems like the biggest hurdle is not understanding requirements the first time around. This is incredibly common.

I work in computational biology, and my normal thought process is that by default, you should expect to write the code three times (especially for less experienced developers).

The first time, you don’t know the problem.

The second time you’ve figured out the problem, but don’t know the best way to do it.

The third time, you’ve figured out the problem and a decent strategy to solve the problem.

With more experience, you can narrow that to just two iterations. But really, especially with research, you rarely have a good feel for the problem domain the first time around. And when you have the expectation that you’re going to throw the code away, you don’t get quite as hung up on implementation details for the first two rounds. And because of that, the process is easier. And you don’t have to worry about refactoring bad code. Just accept the first round as an experiment and take what you’ve learned about the problem to write better the next time.

Reply on news.ycombinator.com