Lessons Learned from Software Rewrites | Hacker News Reader

Lessons Learned from Software Rewrites

alexmartins.me

196 pointsamartinsn84 comments

I've been dealing with a corporate ERP system for some time now, and -- whether rewrite or refactor -- the biggest challenge has been that nobody is willing to assert how it ought to work.

At best, it's "reverse engineer it, tell me what it does, and then I'll tell you what feature I'd like changed."

This is a pretty eternal problem in software development. I now think that it is our jobs as developers to properly identify how things should work.

We're often not experts in the domain we're working on, but by asking questions about a user's workflow on a more holistic level, we probably have more insight into what workflow would be the most productive from a systems standpoint.

Of course, this is a back-and-forth. Sometimes users aren't sure of what they want. And sometimes we're not sure of what is the "best" system for what the users want.

Instead of asking the user where the button should be, ask them when they press the button.

Behavioural driven development (BDD) encourages this process. The developers flesh out specific examples with the business people using readable language, and that becomes user acceptance criteria.

It's no panacea, but it helps because it tries to prevent the business people dictating tech specifics, which the developers take too literally.

For example, a business person says "when I press the submit button the results must be stored in MySQL". What they really mean is that whatever they entered into a form must be accessible later, but the developers now believe they must use MySQL and have submit buttons.

By focusing on the "When I complete the form with data x, when I look at the form again later, then I see data x" you've moved the conversation to the business requirements and away from the perceived technical requirements.

> It's no panacea, but it helps because it tries to prevent the business people dictating tech specifics, which the developers take too literally.

> For example, a business person says "when I press the submit button the results must be stored in MySQL". What they really mean is that whatever they entered into a form must be accessible later, but the developers now believe they must use MySQL and have submit buttons.

So in other words business people are plain old lying about the requirements.

The skill of going from potentially "wrong" requirements from non-experts into something engineers can deliver is one of the most important skills I've seen in technical sales people.

Customers have a problem to solve, and try to ask for something to solve that problem. Understanding the problem and proposing a solution (and a price) is how you get happy customers.

https://youtu.be/BKorP55Aqvg is a great example of how to fail at this. The "expert" totally fails to understand the customers needs, and instead gets hung up on bad definitions. He's supposed to be the expert in drawing lines, so he should be able to understand why they need them. You could say the customer is lying about their requirements, but they probably just don't understand them.

This goes for any customer/provider relationship - internal services, it departments, platform enginners. Not just selling stuff to other companies.

I don't think it's a great example, because with a little creativity that "impossible" spec is possible:

https://www.youtube.com/watch?v=B7MIJP90biM

The customers might even be happy with it, because it would mean they wouldn't have to admit their requirements were stupid.

> because it would mean they wouldn't have to admit their requirements were stupid

I personally prefer it to be instructed when my requirements are stupid (if there really is strong evidence that they are) since this way I can learn in a much faster pace to make good requirements.

> You could say the customer is lying about their requirements, but they probably just don't understand them.

To build on the example from https://news.ycombinator.com/item?id=12428337: If another person says "MySQL" it is MySQL and nothing else. If they meant something different, he would say "data store" or give some non-technical description from which a technology-minded person can reconstruct what the person really wants. The same holds for the "submit button". If the person talks about a "submit button", such a button is meant. If he means something else, he would give a description, say "a user interface element which allows to send the data that I have inputted to the central data store such that [...] Choose the technology which is best according the current UI/UX guidelines.".

If I don't know what I want I give the information in a high-level way such that an expert can reconstruct/build the missing details. If I say some concrete technology that is an exact specification. If this is not what you want, it is lying. There is no in-between.

Lying is when you relate something you know to be untrue. What you're describing is more like "confusion".

Or maybe you don't know exactly what you want so you give examples demonstrating the salient features instead.

Not lying, just not really understanding what they're talking about. For many BAs, 'MySQL' is interchangeable with 'database' so they don't realise that a different database might be a better solution. They're phrasing the requirement in terms of their best understanding of the implementation. The goal of BDD is to focus the language on the business domain instead.

Yes, I've often seen business people pick up on technical terms without really understanding what they mean. It is our responsibility to coax users into giving non-technical requirements.

Sometimes they are trying to deceive people though: Not about the spec, but about their own certitude and level of technical expertise.

> For many BAs, 'MySQL' is interchangeable with 'database' so they don't realise that a different database might be a better solution.

One minute of googling can be expected from BAs, too.

> They're phrasing the requirement in terms of their best understanding of the implementation.

Then they should say "to my knowledge MySQL is a popular database that I know of and is to my knowledge used a lot in the company, but if you know of a better implementation (I don't know much about that - I'm no computer scientist) use it instead.".

This is an opportunity to exercise some social nuance. To them, "MySQL" and "database" are equivalent, and to point out the difference is to make the social faux-pas of overspecification (somewhat of a trope for nerds like us).

By the time you're having this conversation with a BA, the DB is not going to be changed because they made some offhand reference to a different DB. They said something technically wrong, but in a way that doesn't have any meaningful effect on what they were trying to say: as such, to nitpick a detail like that may give the impression of trying to reinforce your superiority by pointing out tiny technical mistakes in their speech. So just let it go :)

Or it's the developer's job to ask "why do you say that?" - the BA thinks they're being helpful and shortcutting a decision for you; the least you could do is try to be helpful back.

> the BA thinks they're being helpful and shortcutting a decision for you; the least you could do is try to be helpful back.

Giving an exact specification is a statement that you want it this way. It is a property of an executive position to give correct instructions to the subordinates. If they are not capable to do that they are simply not qualified for their job. On the other hand: If you want feedback for your specification, ask for it. Giving instructions means "you implement it this way and I am personally responsible with my honor that this is indeed what is needed".

You seem to have a very rigid view of the business world. That is fine, but you are very likely to find yourself to be more effective if you can adopt a more flexible outlook, where you see peoples' competency in their jobs as a continuum rather than black and white, and where you see instructions as conversations rather than rigid honor-bound orders (unless you're in the military). Many have tried to make the organizations around them reflect their rigid prescriptions for how things should be done, and they mostly just succeed in making people dislike working with them.

Maybe it's like that where you work. But if I was the BA and was having to be personally responsible for something I would want to make sure that what I was specifying was actually correct by discussing it with the people who will be doing the work. Because that responsibility cuts both ways.

It would make developers (or anyone doing a job for someone else) lives substantially easier if people behaved the way you think they should, but the reality is that 99% of the time they aren't doing something that most people would consider to be lying, they're just being people.

> Behavioural driven development (BDD)

https://cucumber.io/

is a particularly awesome idea for specifying behavior. really want to try it out for a project sometime: the idea is to write the spec in these domain-specific "gherkin" files.

i find it comparatively difficult to get get buy-in on collaborating on anything machine-readable, however, unless it's from people who are comfortable with general-purpose programming languages, anyway.

nonetheless, in an industry where it's fairly standard for designers to build in domain-specific languages, i don't think it's an unreasonable skill for product managers to consider acquiring.

any user experience stories from cucumber, etc. on rewrites or otherwise?

In my experience it's a joke. It's a way for devs to write executable document strings for the tests they also write.

Absolutely no business person who is responsible at the level BDD works (the head of marketing for example) ever ever ever writes a working BDD description or frankly bothers reading them.

Maybe just maybe they are useful intermediaries for Business analysts but most of those have been fired.

Fitnese tests have more traction but then this is just exposing your API for user tests

This matches my experience. I used to somewhat blame the business folks for being too lazy and/or pompous to bother with their side of the bargain, but I've come to think it's just generally a solution to the wrong problem. The hard part is correctly translating from what is asked for to what is implemented, and executable requirements don't have anything to say about that step. In cucumber, I can take any arbitrary Given / When / Then statement and implement it in any way, including ways that are completely wrong and (worse) ways that are subtly wrong. The translation code just becomes yet more code that may or may not be correct, and in my experience, it can be a lot of code. When you find yourself wanting to write unit tests for your acceptance test pseudo-english translation layer code, you're deep in a rabbit hole.

i think this (the parent post) is truth: i've had conversations a/b behavior specification where edge cases simply aren't taken into account. so the specification would say, for example

"it does this when anyone clicks a button."

whereas there's often some implicit question, like

"what happens when two people click the button at the same time?"

and that kind of concern -- where the logical implications of requirements -- is not always handled by top-down requirements management... i'm almost certain i saw a fowler article on "conversational" requirements management recently, and that made a lot of sense.

In general, BAs should not be writing the stories, but they should be able to read them together with the developers.

I find that it helps, but it doesn't solve the whole problem. The most useful bit is the stories with examples, as those are what are so often missing from specs.

The problem with edge cases is that they are infinite. If there's an edge case that's likely to happen (ideally based on evidence of past behaviour), then it can either be in the spec or just in functional tests.

I like to think that BDD is like writing the documentation first. Would you put the edge case in the documentation? If not, it probably isn't the BA's responsibility to decide what should happen, and it shouldn't have a Gherkin-style story.

But it's not a silver bullet. It's a tool that helps solve some problems on some projects in some teams. Usually, it helps in cases where the domain experts are not very technical and where the domain problem is a fairly simple set of business rules.

This is my experience as well. Cucumber and the like sound like a way to get 'the business' involved more, but in practice the cucumber files just become yet another thing that the developers or testers have to write. At which point you are better off with more traditional testing methods.

> the cucumber files just become yet another thing that the developers or testers have to write

yep, if (and it's sounding more and more like "when," even outside of my anecdotal experience) that decision is made, there's not much point in using cucumber or similar.

i was kind of hoping someone would say, "yea, our product manager writes domain-specific specification files, and the workflow is so much easier!" ... but no, i'll just continue being mildly cynical a/b the role of developers in industry.

I'm more of a PHP developer, so I use Behat instead of Cucumber. However:

1. The value of BDD is that you have some stories with examples written in the language of end users. You can use a tool like Behat or Cucumber that parses those stories and runs them as automated tests, but it's not fundamental, and you shouldn't avoid using BDD just because the automation aspect is too difficult.

2. Avoid using Behat to interact with the browser, make HTTP calls or run command line scripts. Otherwise your features become too tightly coupled with your views. Use Behat to test your services and domain entities directly, and not to test the framework, because testing the whole application (including the framework, if you're using one) is slow and wasteful. If you want to automate the browser, there are better/faster options than Cucumber and Behat.

3. BDD features are not the same as functional tests. Your features should focus on the typical use cases and read like end-user documentation, while your functional tests should include edge cases and should be optimised for developers.

Negative experience here: someone tried to use it as a silver bullet and it made underlying problems worse.

Our "tests" are all written by "tell me what to click" QA, and the Behat source is literally commented-out because they cannot be run by programs, only the QA who made them.

mysql isn't a great example because that frequently comes up specifically because they have db analysts with skills in that system, its where the rest of their data is, already have support contracts for it, etc.

While that might be a valid reason to require MySQL as a database, it's probably not a requirement of a given form. It might not even be necessary to have a database at all - it's important to understand 'why' the BAs are saying this, and not take their opinions as the definitive solution.

For example, the ticket might look like "When I submit the form, then the DB Analysts are able to run reports on the submitted data".

The decision about what database to use still isn't a business requirement in that case though, more of a nonfunctional requirement that can be determined outside of a particular feature.

It very much is. If you told them you might decide not to use mysql so it would be completely incompatible with all of the data storage requirements, they would rightly tell you to get out for wasting their time. Your refusal to use the correct storage could cost them thousands of hours of labor establishing new processes for backups, BI integration, high availability, and just training on managing the system.

I have worked with many companies where this was a very hard business requirement. Companies that dont care how the data is stored tend to be pretty small or not actually care that much about the data.

"And I need you to write it in PHP because it's cheaper"

(not hating on modern PHP, just its reputation and 'outsourceability')

This is my life. Replacing software that was written a decade+ ago so that the software has become the process. Users find is down right difficult to think in terms of what is going on a business process rather than the steps they have to do to force the computer to say yes. Many find it genuinely tricky to even describe in even real world terms what is going on without slipping into jargon that is actually just describing the existing system.

I'm currently doing the same and have found a lot of my time is spent interviewing managers and their reports to devine what problem they are solving with the software.

In general they are completely unable to describe the job and problems they solve for the company, only how they use the software day to day, with out significant coaching and reframing.

It's a vicious cycle: Nobody knows the vision, so they just iterate on the process, and the process is inscrutable, so nobody can extract a vision for it.

This is why iterative development is so important. Most non-tech people can't explicitly state what is best for them. Half the issue is not knowing what's possible. Half the issue is not being wired to be so explicit.

This does out some burden on documenting what you did after the fact, but it beats the method of locking yourself into something that's wrong to begin with.

The developers who can get enough domain knowledge to intuitively understand what's needed add enormous value. They can credibly suggest better ways to operate. I refer to it as the second 10X in productivity gains.

> Of course, this is a back-and-forth. Sometimes users aren't sure of what they want.

Or your "product owner" doesn't really know the product and happens to be in a minimal-overlap timezone, and walking to the other side of the local office to ask Real-Users for feedback eventually generates a subtle reprimand about going through proper channels...

Yeah, not planning to stick around.

If I look back over my career the projects where I had direct access to knowledgeable users were always easier and probably more successful.

One of the red flags I started to notice was where the client liaison had only been working there for a couple of weeks or months.

In an ideal world, instead of asking the users anything, sit behind them and watch them work for a while. I've found that I learned more about what people actually need that way than any question I've ever asked.

I would say that's one of the reasons why automated tests are so important.

Awww, hell. This is my life.

When I hear about rewrites, I always think about this comment:

https://news.ycombinator.com/item?id=11554288

I expected some wisdom, but that comment basically says that a rewrite either isn't needed because programmers are already excellent in the first run, or the rewrite will end with a similar shit code in paraphrase, because programmers have learnt nothing. Also it takes time, which costs money...

This is probably going to sound weird, possibly cheesy, but cottonseed got something out of that post, possibly more than I intended to convey, so I hesitate to follow up for fear of taking that away.

What I think I was getting at then, and certainly believe now, is that some people grow with the code, learn from many of their mistakes, and do the long quiet job of expunging their worst errors as they go. Their code is the best proof that they could accomplish a rewrite, because it already is a rewrite. They just didn't need to stop the world to do it, or at least not all at once. Perhaps a month here and a couple weeks there.

For the rest of the time it has been like the Ship of Theseus. All of the parts are new, and yet it is the same ship.

Meanwhile the rewrite guys are continuing with their mess and hoping they get a do-over, putting off things hoping for Some Day. And on that magical day, all of the bad habits of the entire team (management included) will correct themselves overnight.

Bad habits are too hard to break. The best of us know that you don't abstain from bad habits, you crowd them out with new, better ones. But that takes time, practice and determination.

If you want to condense it down to nothing new under the sun, that's fair, and I would offer that I wanted to believe the rewrite folks were onto something, but at the end of the day, breaking down the problems and using relentless refactoring seems to be the only thing that works, without changing the definition of success to achieve victory.

Additionally, as time passes you get new people working on the system that does not understand it as well as the original designers/developers, simply because they already have the system and their knowledge becomes pigeon holed inside of it. They don't have the original scope.

And then to top it off; if you did rewrite the system you not only need to reverse engineer N years of code, but also N years of bug fixes and random miscellaneous snippets that made it into the system and that no one who still works at the company understands the purpose of any longer.

And like the Ship of Theseus, it help the shipwright a lot when the guy who owns it is rich enough to check everything thoroughly before going out to sea. :p

edit: Yes, no problem, i understood now what you meant, crdoconnor put me to it.

No it isn't. It says that good programmers are doing mini rewrites all the time negating the need for a big bang rewrite whereas bad programmers are probably just piling the debt up while asking for permission to rewrite and will do once again when they rewrite.

There's some truth to this I think. Big bang rewrites are best avoided where possible and there's plenty of ways you can incrementalize refactoring that don't occur to most people.

I think you're right and i misread it. So if the mini-rewrites in silence equal relentless refactoring, that might equal a rewrite nobody ever observed, excellent.

Sometimes a better approach is to wrap the system you want to deprecate rather than have a parallel implementation. First, because you might not fully comprehend the current implementation to replicate it. Second, because in real life often there's not enough time, budget and coordination between teams to migrate old clients.

Since their main requirement was having a REST API for new clients, and apparently the old implementation worked fine, they could have implemented a EJB -> HTTP layer on top, and have new features talk straight to the DB. Eventually, they replicate all the features talking straight to the DB (w/ the added benefit they can test against the codepath going thru the old system). Finally, death by phagocytosis.

Unfortunately that doesn't work either. I have seen libraries with 5 (five!!) levels of wrapping because each developer that wrote a wrapping level didn't understand the layer below/thought it could do a better job with the interface/was prematurely generalizing.

I think the problem was, that they had to implement new features for the clients only supporting the old version, too. At this point you are left with a HTTP wrapper around your old software, which does not qualify as a rewrite.

Well, yeah.

It sounded like an HTTP wrapper is all they actually needed. Or in fact any well specified wire protocol with lots of implementations. It could have been protocol buffers over raw sockets also.

Strangler pattern FTW, well that and never have two systems share the same database. Always have one source of truth.

The database is the source of truth.

A great pattern is to enforce consistency in your database. Rather than e.g. adhoc code in an ORM.

Could be worse. Imagine a situation where you have to work with an old Java code base, where there is a lot work arounds that had sense ten years ago but not any more, and you are using obsolete library versions like Hibernate 2. Add that you can find similar functionality made on totally different ways. Add that you can find abuse of some programming patterns. Add that you have your own XML based framework where you end doing XML based programming, and not sane way of debugging it except doing the equivalent of debug with "print("CACA"); return -1". Add that for some operations, the java code must do the same stuff that does a heavy client wrote on Visual Basic 6 that works directly against the same database. Finally add that the database design isn't the best (BNF form ? What are you talking? )and that you are stuck on using a internal tool that does the same thing that Liquid base, but worse, and it's based on some old abandoned Apache libs that no body knows well how are working or how we modify. At least, we know that we have a problem and where are trying to fix it.

And this is my diary work.

PD: I have a friend that lives on a worse situation, where they have an application based on Java 1.5 applets, with bad or not existent documentation, and nobody knows how works internally.

Are you also Brazilian like the OA? I ask because I know diário means daily in portuguese, but in English "diary" means a personal journal and doesn't make sense in this context.

Spanish. Thanks! I should wrote "daily", not ?

Yes, indeed!

Our scenario looked a bit similar to yours mate!

TIL:

>The strangler pattern is a software architecture pattern where a legacy system or feature is strangled with new code.

>Make something new that obsoletes a small percentage of something old. Put them live together. Rinse and repeat.

Strangler Applications http://paulhammant.com/2013/07/14/legacy-application-strangu...

In my opinion, asking for "removing unused features" is one of the consequences of full rewrites. Also, a reason for having to maintain "dual systems" for years. It is not infrequent having "dual systems" and having to deprecate the "new", just because after e.g. 10 years you can not get rid of the old, or even worse: the "new" performing worse than the old (e.g. old simple centralized system working much faster on new hardware vs "new" pseudo decentralized mess that can not scale as good because inter-node communication overhead).

BTW, every time I read Martin Fowler quoted in a post as full rewrite justification for the "faith jump", I get scared.

I don't know why the poster didn't implemented the new API, using it for the new features, and eventually, rewrite old API commands (one by one, not all at once) calling the new API, so in the end, new API would remain, and old API would be just some wrapped calls to the new API.

That was exactly our approach, for the old HTTP API, we kept the same contracts (URIs, response body, etc.) For the EJB one, as only one remaining client was using it, we deprioritized that, but we were going to pick the features they were using, and make the Java client point to the new API implementation, also keeping the same contract.

Joel Spolsky has a very nice writeup about the subject, "Things You Should Never Do, part 1":

http://www.joelonsoftware.com/articles/fog0000000069.html

From the guy who had a new language invented instead of re-writing something...

I think Joel's articles were probably best when they convinced you of something you knew but hadn't articulated yet. If you aren't there yet I don't know that he always delivers.

And if he's selling an idea you've already abandoned, he can leave you flat for sure. His strength is in introductions, not reconciliations,

Binary advice is rarely useful, you should never rewrite, you should have 100% test coverage, you should only hire graduates and on and on.

But the siren song of the from-scratch rewrite is so difficult to resist.

I think of this as a saner version of the from-scratch rewrite, personally.

it's difficult to get a picture of all the nuances of this project, of course, from any quick read.. one thing that the article left me wondering a/b is the outline of "the new stack" which is "agnostic to any programming language".

i mean, i'm not the biggest Java fan, but it's alright, and there are some pretty good RESTful frameworks

https://www.playframework.com/

while the strangler application (always a fowler link, always, in these types of articles) makes a ton of sense for making sanity out of incoming requests, i wonder if some effort could have been &/or was saved behind the transport layer by reusing (possibly with refactor) the existing Java implementation?

whether doing so would make sense would be really situation dependent, of course, and an "outside" perspective from a quick article isn't going to have enough information to say anything conclusive. the (lazy) feel is that reuse is better than duplication of effort.

------------

aaaaanyway, super-good article. this is of of the kinds of things that's important to think about as a workaday web dev.

Hey, thanks for your reply. Your line of thought is very similar to what we had. We considered refactoring the existing Java implementation. It used a really old Jersey version, with too many overengineered customizations tightened to that version. We tried to upgrade it and get rid of some of those customizations, but the system was so fragile that other parts of the system stopped working, which made that a really difficult road to drive. We wanted to simplify the design. We chose Scala and Finagle for the new stack.

> So once a functionality was implemented on the new stack, with all characterization tests passing, all we needed to do was to tell the router to use this new implementation, instead of the old one, and everyone consuming this endpoint would be automatically using the new implementation!

How would that work without also migrating the database and while keeping the databases of the old and new implementations in perfect sync?

Migrate the DB in a backwards compatible way. So both version n and n-1 can use it. The next update (n+1) can then remove support for n-1.

This isn't a rewrite in the traditional "big bang" sense of a rewrite but normal software evolution and maintenance.

Anyway sounds like their approach was quite reasonable, just a bad title for the blog entry.

It's an incremental rewrite. Rewrite one feature, switch everyone to use it, then destroy the old implementation. Do the same for all other features.

to grasp how bad that system agree, globo is Brazilian fox news. and only got on the web 10 yrs ago... and they're already on version 4

Great article! It's interesting though, how did you convince management to remove features no one was using?

From my PoV : Pull the plug and wait to see if someone yell. If noone yell... then noone need to know.

If someone yell, be sure to put your own contact proeminently everywhere. So that your management do not know. And just roll it back

Problem with this is the feature that is used by some external actor once every 18 months. You'll pull the plug, and when 6 months go by, you'll think you're safe. Then when shit blows up, you'll have no context and spend 3 days tracking it back to the "pulled plug"

If no one was using them why do you need to convince management?

Because management decisions ain't always driven by logic.

The simplest reason may be attachment to the feature--the feature is of no use but the person who's invented it is scared by the possibility of removing it.

Because when trying to sell the application list of features is longer, which means we know what we're doing and are able to maintain it.

Reply on news.ycombinator.com