Ask HN: How to escape the vicious cycle of technical debt | Hacker News Reader

Ask HN: How to escape the vicious cycle of technical debt

16 pointstuco8610 comments

At my company time is always running short which is the excuse for all kinds bad or short sighted decisions, IMHO. Obviously this slows us down and makes time short again. I try to raise awareness of technical dept and common anti patterns, but i don't come through. I talked to some of my coworkers and they see the same, but all of us seem unable to change a thing.

I wonder if anyone can give me directions on how to improve things. Especially with the product owners/managers that have less technical background.

Or maybe a piece of good advice on how to deal with that personally. I also considered if I was the problem - by somehow misjudging the situation or something - but i think like myself, so don't spare the criticism.

---

A current example of the type of problems we face.

A big move from our DC to a private cloud based solution is coming up because our hoster closes that location.

My coworker and I where charged with building the new infrastructure. We built a sweet CI/CD infrastructure using gitlab and kubernetes. We did in ~5 months which took ~10 devs twice that the last time. So things are improving actually - yay! All our newer services where easily migrated, new prometheus monitoring is a treat. There is the last big monolith tho - two people and a short hard deadline while maintaining all the other services. So we sat together with the relevant devs and decided to fix away all those doodads in nginx and varnish configs and make the software usable on the new infrastructure together.

Now product owners start to complain that their tickets don't get completed and some even started to pull rank and moved devs to other tasks - moar tracking. They seem unable to grasp that a site can actually fail when load can't be handled, so they don't get stressed like i do.

Currently the site takes ~20 seconds to render at some places. so there is no urgent need for improvements? I don't get it, should i just watch it fail and relax some?

> Or maybe a piece of good advice on how to deal with that personally. I also considered if I was the problem - by somehow misjudging the situation or something - but i think like myself, so don't spare the criticism.

Before anything else, I really like and respect your attitude. As a developer, I would like to work with more people like you.

Further, as a developer, seeing an application take 20 seconds to render makes me physically ill. I would feel exactly the same way you do, and ask the same questions.

The 20 second rendering time needs to be fixed. But, in a situation like this, there are always two options. It should be the first priority, or other things are higher priority.

Sometimes, I think it's good for developers to look at a business as a predator. It doesn't particularly care about anything other than staying alive. It has to take in as many or more calories than it expends or it will die. And, in order to catch that many calories, it has to be ruthless.

Businesses are like that. If a business consistently spends more than it makes, it will eventually die. It has to maintain positive cash flow over a certain period, no matter what. (That isn't completely true all the time as tax planning can come into it, but just accept this as a helpful half truth.) This means that when it comes to fixing problems, it's not about fixing the bullshit, it's about fixing the most expensive bullshit first.

A 20 second render time is fucking atrocious, but maybe, at worst that will cost $5,000 a month. Whereas, maybe there is a feature that will bring in $8,000 a month. In that case, the new feature wins, technical debt be damned.

All isn't lost though because again, you have a good attitude and I'd bet that other people notice that too. If you are prepared to lose graciously, you can bring up your concerns to management.

However, it's important to speak their language. They likely can grasp that a site will fail under load, but they might not care. They might not care because either something else is more valuable to work on, or because they don't know how costly your problem is. If you want to win this argument, put together some numbers and go at it. But, always remember that a starving lion isn't going to pass up a sick gazelle because gazelle meat gives it gas...:)

thanks for your thoughts.

Numbers seemed really valuable to us too, so we made shure to bring our metrics up to speed.. but you need technical skills to interpret those as well. Now we have - a lot of - numbers, but reporting those is hard, time intensive work too. We keep trying tho.

That's a great analogy!

I want to suggest a related tactic, which is to relate your problem to a secondary goal that another part of the business has.

For instance, your marketing team wants your site to be ranked in the top 10 on Google for a certain key phrase where it currently ranks #100. Marketing can probably quantify the value of that improved ranking. But Google penalizes sites that run slowly.

Not to mention the studies that show most users bounce if a page takes more than a second to load; and a good marketing team will consider your (presumable) bounce rate atrocious and be trying to improve that.

The marketing team can be your ally in the fight for resources to fix this problem.

What's the cost per hour in customer service hours and in lost sales if the site goes down entirely? Quantify the likelihood of that outage based on current traffic rates. Recruit the customer service and sales teams to your cause.

People here have already given good advice how to deal with product owners/managers. So I'm rather going to talk about the root, which is how to deal with technical debt in the first place.

As a developer myself, I know it's hard to review the Pull Request with good insight just by looking at the diff. What I want to say is that we sometimes might neglect some technical debts by not understand fully the high-level overview of the Pull Request. So in order to avoid that, you and your fellow devs might want to spend more times to review Pull Request and talk about the technical debt that will be introduced by those PRs. This will vastly reduce the amount of debt you have to pay later in my opinion.

Apart from improving it in a normal way, there's a tool for that called Softagram. What it does is that for every Pull Request you made it will analyze your PR and then post a comment with high-level overview graph (or picture) about that PR directly to your PR thread. So that it's much easier to discuss with your fellow devs about what decisions you should make regarding the technical debt. As a result, the process of reviewing PR is much more enjoyable, faster, and more secure. You can check out the tool at https://softagram.com

In the ideal world that's how it should be, in every pull request, the reviewers should check the architecture and implementation to fix issues increasing the debt. Instead to be fixed later with 10-100x costs.

Hi, thanks for your thought. You might want to checkout how Softagram can help you here: https://www.youtube.com/watch?v=_3OzOVIOmkQ&feature=youtu.be

> ...using gitlab and kubernetes...

It's not technical debt. It's recent, trendy and slow stuff. It seems some bad decisions have been made. If you're asked, start by saying that. You need to understand why it's bad before by yourself.

> Currently the site takes ~20 seconds to render at some places.

Make something custom to solve each case. Pick what you think is the fastest this time (not the trendiest). The fastest is static html when it's possible.

You should read "The Phoenix Project", I thinks it explains very well how you could get out of some of your problems.

Can you add the high-level version of that explanation here so other HN users have an idea what it is about, and decide whether to check out the book?

two devs and five months sounds like a waste of maybe $100k. Yes I would not touch perf if no one is complaining about it. On the other hand product "owners" routinely drive companies into bankruptcy.

Reply on news.ycombinator.com