Undebt: How We Refactored 3M Lines of Code
engineeringblog.yelp.com
engineeringblog.yelp.com
This feels relatively safe in Java because tooling can staticly know a lot about your code (and can know for sure that a particular call site is the method or class you're targeting). I've be terrified to do it in python without a very thorough test suite.
[0] https://www.jetbrains.com/help/idea/2016.2/structural-search...
But python grew up in an enviroment where this sort of thing was not practical, and the batteries included is actually a good approach for what python tries to do - scripting. It just doesn't scale well in to maintainability.
The solution to this, of course, is sandboxing - never install libraries globally, only on a project-specific basis, and then their dependencies can override global ones. But it's fiddly to get the UX right - in Python, managing that involves two separate tools. And you'd need to create a project directory to get a repl with some library in it, unless you had extensions to the repl to install things temporarily - and then you'd likely have two separate UXes for installing things, whether you're doing it for a REPL or a project.
It's a bad idea, on short timeframes. But it's a hell of an adaptability bonus that ensures the language will keep improving.
Just like Haskell's extension system, or its loose dependency on Prelude, or its multiparadigm emulation.
I keep hopping something better than Haskell appears and people move on - but it probably won't happen any time soon, as soon as something better gets traction, Haskell will simply devour it and keep growing.
Backwards compatibility is mostly an attitude problem.
Refactoring in python is bad even if you restrict yourself to "sane" code (no metaprogramming and abusing dynamic stuff so the tools can follow what you're doing) if you need something that will work for everything out there it's just impossible to do.
The impact of backwards-incompatibility is often worse than you expect. If anyone should know that by now, it's the Python core. Or perhaps the folks that faithfully waited for Perl 6.
Can we please stop putting forth this idea that features >>> reliable product? The amount of dev time that a company will save from removing technical debt will likely be more than the extra sales the company will get from a new feature. I look forward to the day where the executive team comes to the developers and ask why they are working on features instead of cutting down technical debt.
Not always true (though it often is!), but particularly not always true in the timespan that the company needs to get sales in...
It might just affect development time of things interacting with the bad, old code - or make bugs harder to fix Changes to old nasty code will probably be hacky under the mentality "we're going to refactor this anyways, no point in writing clean code /now/"
Most old code (i've worked with) will function fine - it has for a while after all.
<google google google>
Here - pont 5 in this post: http://www.joelonsoftware.com/articles/fog0000000043.html
It's hard to explain to someone who hasn't experienced it how much easier it is to develop in a bug free code base.
To paraphrase: We programmers need all the help we can get, and we should never assume otherwise. Careful design is great. Testing is great. Formal methods are great. Code reviews are great. Static analysis is great. But none of these things alone are sufficient to eliminate bugs: They will always be with us. A bug can exist for half a century despite our best efforts to exterminate it. We must program carefully, defensively, and remain ever vigilant.
It's a great philosophy or methodology, but it can be very hard to execute on 100%. In the real world there are deadlines and resource requirements and impatient clients and produce launch plans with set dates and trade shows starting next week and the Thanksgiving/Xmas launch opportunity and and and.
Better languages are possible. Provably correct software is possible. We really can eliminate bugs.
Ah cool, so you've solved the halting problem then?
It's like the difference between being able to express proofs in maths, versus having a method of deciding the validity of arbitrary propositions. We can certainly write programs that we can we can prove things about, but given an arbitrary program, we can't always tell whether it will do the right thing.
Even then, the software is provably correct according to some specification, which can still have bugs. No technology can fully prevent logic bugs.
At other places I've worked, I wouldn't raise an eyebrow if I found 3 bugs in a day, just trying to get other stuff done.
> As soon as you see a bug, you fix it. Do not continue on. If you don’t fix your bugs your new code will be built on a buggy codebase and ensure an unstable foundation.
Looking back to the codebases i've worked with, this advice seems extremely wise.
[1] http://www.gamasutra.com/view/news/279357/Programming_princi...
Improving and perfecting the code is a task that can take infinite amount of time.
At core all of Agile Software Development is aiming to mirror Lean in the manufacturing world. At the core of which is the concept of minimizing work-in-progress. Because WIP may result in waste if you subsequently find a problem that renders the WIP null and void. Hence the drive for continuous flow to minimize WIP and a "stop the line" mentality to spot and address issues when they arise.
I'm often amazed at how many people "doing" agile through sprints don't understand that the sprint is just an artificial mechanism to minimize WIP.
Not dealing with technical debt when it occurs is just increasing WIP.
I had a Philippino friend who told me of his early experience in a Japanese development outfit - the team got praised by management (maybe not CEO) on performance improvements and code reduction. At the time I thought those priorities would never have flown in the US (though was of like thinking).
So this mindset does/did exist.
If are building something that will be sold, particularly in a competitive market, management will almost always believe that new features trump reducing technical debt. To a product manager, even four weeks of technical debt reduction sounds like "no new features for a month."
Yeah, but if you can't implement new features in under a month because the code is such a mess, then maybe spending a month cleaning it up will allow you to do more features in a shorter amount of time, thus increasing revenue faster than just hacking more features into the existing crappy codebase. Obviously it depends on the codebase, but you have to look at both long and short term goals to make good decisions.
Very new to talking with execs and I don't know where to begin.
Does anyone here maintain such large codebases? Are they truly that big or are people just counting third party code and generated stuff?
It was code duplication. One example I saw was a many thousand line css file included 4 times in a row. They were all different generations and some were customer specific so it was an impossible task to try to consolidate.
That was a common theme. Enterprise stuff where one customer needs slightly different behaviour so code is duplicated (maybe thousands of lines).
The first time you take that road, you've doomed yourself. With a good / small team they can manage it- but imagine when you've got 20 devs sitting there of varying experience. The precedent is set and the solution for anything mildly complex will forever be, make a copy.
switch(clientId){
case 42: //todo: Make sure database agrees
...
}
Because the "risk" of actually changing code for any client where things already "just work" is a testing risk, and testing is nearly always the bottleneck at these places in my experience, if they've even managed to hire a dedicated tester yet.
Of course if you apply the Joel Test you're likely avoiding these places. But they can be rewarding places to work.
That's stuff you don't have to hardcode -- you can pull it out into a 'rules engine' (at the expense of an additional runtime dependency) or push it further down into, say, database stored procedures (I can hear some of you shudder). But for us, the rules rarely change, or change at a pace that's acceptable to keep up with.
Also, there are a lot of views and specialized interfaces tailored for particular workflows. In several cases, the data underneath of them is the same, but there are different UIs -- that adds LOCs considerably.
If your application language also does a decent job of expressing the business logic, that's great.
May be when you have rules that change every hour back and forth.
If you look at Yelp and see a simple crud app you're not looking hard enough.
However, if you require that, you'll notice that very little data manages to make it through your entry port. Few suppliers will want to bend their system for you to comply with your rules, and those who do will find inconsistencies in your specs or just ignore them (different encoding, not sending state abbreviations separately from city names, forget to encode HTML entities, encode them twice or thrice (my personal record find here is four levels of HTML encoding in data stored in an excel file), etc)
So, you end up with quick and dirty hacks that convert the suppliers format (or, rather, what you _think_ it is, as the supplier won't be able to tell what their format is, either) to yours, fix some egregious errors, etc.
Hundreds of data suppliers, code with tables mapping their codes to yours, or that map known errors in their input to corrections, 'smart' programmers who notice that they can replace that 10,000 entry table with corrections with 20 lines of code and a 1,000 entry table, except that, a few months on, that 20 lines have ballooned into 1,000 that nobody understands, so that this code can't be used for handling the data from a new supplier anymore, and the line count starts increasing rapidly.
On top of that, once you operate world-wide, you'll learn the joy of differences in addresses. Does a country have states? Zip codes? If so, where does one specify them in an address? If you want to localize that in your app (in yelp's case, people may want to show an address to a taxi driver. For that, it would help if the address followed local conventions) line count skyrockets.
This is not the first time this issue has been faced. Why would they reinvent the wheel instead of just using libraries and conventions?
Also, assuming a robust generic solution exists, it will almost certainly be slower than a customized one. It is very tempting to think "but our data is relatively clean, we don't need the full feature set of that library that makes imports take an hour longer to run".
I've commonly seen codebases explode the 1M LoC mark. Not necessarily for a single component, but if you have multiple systems interfacing with each other it's really quite common for business applications. That's obviously excluding libs.
Frameworks are a bit responsible for this in my opinion (note: not saying frameworks are bad, but it's a side-effect), as you'd often either have some additional configuration, boilerplate code, or generated code.
Also, it your application is long-lived (think decades), it's even less surprising: new engineers come and go, and it gets harder and harder to touch the things that were maintained by the previous key-holders... So you had a new stone here and there, polish a turd here and there, but you don't really untangle the mess that sits right at the middle, because it's just way too dangerous (or so you think). And it goes on and on. And it's aggravated by the fact that, as the project grows larger, the barrier for entry for new developers get higher: it takes longer and longer to understand the system/platform in depth, and many never even try to get there.
Processes and security concerns affect this too: you're often only allowed to fix something which has a ticket assigned to it, originating from a business user. Touching anything else is a big no-no, as it would mean QA has to re-test all the things that could be impacted. Of course we can argue whether that's actually the case and how proper testing would mitigate this, but you see my point...
This code might also include HTML templates and CSS or JS files that can easily grow large.
It feels like applications grow with the team that develops it. At our peak, we were around 10 people working on it, and it's just so hard to do meaningful refactoring when large parts of the codebase are being modified by 10 people in parallel. Not only the technical challenge, but also getting everyone to agree on and embrace patterns and structure.
Also it's easy for business apps that have been around for 10+ years and continually worked on during that time to get to 1m-2m+ lines of code... and jump from 2m to 3m is easy too.
Of course not all lines are code. About 140k lines are flat file database of configurations, and the other ~300k lines is code/templates.
300k lines of code to manage 14k applications. lol.
------------------
Actually did a more thorough analysis cleaning out comments and what not. The database is more manageable ~86k lines and template generation code is around ~47k lines. There's a crap ton of whitespace and comments.
Just to get some real data on what works, rather than anecdotes from veterans.
Clarification: what I mean is there is a lot of ad-hoc stuff in these blogs that may be narrowly true or applicable to a given project or subset of a domain, but generally there is either significant evidence against some of these blogs' contents or the contents themselves have absolutely no formal, rigorous support.
Nice analogy btw.
[0] https://homes.cs.washington.edu/~mernst/pubs/mutation-effect...
Seriously, this sounds like exactly my kinda gig.
With 3M lines of code gone, it must be terrifying to feel that it may have broken something. How did you ensure that it is still working as before?
Edit: Grammatic corrections.
The question was within a somewhat specific context (team of scientists, visual programming environment, etc...) but I use the same approach for all projects. Never reached 3M LoC for a single projet or system component though. More like 1.5M. Anyways, that process works for me. Maybe it does for others.
https://msdn.microsoft.com/en-us/library/dn269218.aspx?f=255...
This is paraphrasing of Chairman Mao:
"The policy of letting a hundred flowers bloom and a hundred schools of thought contend is designed to promote the flourishing of the arts and the progress of science"
And the ripping out by roots part brings labor camps to mind:
"After this brief period of liberalization, Mao abruptly changed course. The crackdown continued through 1957 as an Anti-Rightist Campaign against those who were critical of the regime and its ideology. Those targeted were publicly criticized and condemned to prison labor camps."
Coccinelle is used extensively by Linux kernel developers for a whole tonne of things like this.
* Double quoted docstring
* Remove unused imports.
These things are largely cosmetic.
This is the wrong way to think. Refactoring will save time by making code faster, more reliable and making it easier to build those new features in the first place. Looks like their biggest issue is bad technical management, not deprecated code.
The admin and back-office system is deeper. Beyond the reviews you also have: events, mailing list management, profile management, messages, i18n, search, etc
Example: http://engineeringblog.yelp.com/2015/10/how-we-use-deep-lear...
Edit: still, 3Mi lines is massive. However, I think there's something that contributes significantly: HTML and CSS
The biggest reason for the bloat: People are not allowed to change code. You only add code. Now try to add a button to some GUI without changing a single line of the existing code. At this moment I understood, why enterprises produce those overengineered AdapterFactorySingletonDecoratorBridge stuff.
Ply tries to implement lex/yacc only, so it's far from a nice interface really...
Or specifically: it's trivial to match `specific_func(any_expression, {capture, these, values})`, but it's practically impossible with regex and painful to post-process with ply.
Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.