Comment by different_sort | Hacker News Reader

Comment by different_sort | Hacker News Reader

Parent Full thread

I work in an huge enterprise. We have incredibly customized software and stacks that have not changed much for 30 years, because they did not need to.

Now the people who wrote those stacks and who understand them are retiring/quitting. Kids coming out of school don't want to learn these systems, nor do people off the streets. You can only pay people to come out of retirement so many times to keep the plant running. This is above and beyond mainframes, and is intertwined deep in the code that powers every single application that runs the plant today.

We can't run off the shelf software on-prem, a huge level of customization is needed to bring it in.

We cannot pivot quickly to new things or support new languages.

We really struggle to add new features/releases and add new software to drive revenue. The IT overhead that just goes into keeping the plant running every day is astounding.

This is what I think of when I hear technical debt.

Going down this road did give us advantages for a long time, but now we're in an enormous crisis. It's not an insurmountable challenge, but I would be surprised if there aren't a lot of large companies who are brought down by their technical debt as faster moving competitors move around them. I certainly feel that unless we get our act together, we will be disrupted.

I found this response surprising.

Code that has worked for 30 years is more technical debt than the ability to support 'new languages' or 'pivot' the code? Maybe I am an old fart, but the opposite seems right to me.

Large code bases are difficult to add functionality to.

A code base that is easy to pivot and switch languages sounds more like a nightmare. Isn't it more likely that your new, zeitgeist-capable software would torpedo development in far less than 30 years. As I understand "new programmers", the 'reinventing the wheel' speed is measured in a few years not decades now.

Or did I misunderstand you?

I think it represented poorly. It's more than just code in one system. It's systems built upon systems built upon systems. It encompasses our network, our software deployment stack, our proprietary extensions to standards and much much more. Unknown dependencies on unknown dependencies on unknown dependencies (and it's not like we're slacking on trying to map that/keep the asset inventory up to date).

It's basically paralyzing. It's so hard to get a release done, add capacity, or add new features for our lines of business (we have dozens!).

How do the execs plan to handle the problem?

How aware are the execs about the situation? Or is it primarily the engineers, maybe technical managers, who can see the problem currently?

(Is it ok if I ask what's your job role?)

Job Role/Further details are risky to discuss because this forum is read by my colleagues and likely the nerdier execs. I could leave it at I am someone very senior and actively involved in trying to tackle our problem, so I see the efforts and the challenge first hand.

I can tell you that execs are very aware of the problem. Higher ups have spoken about it at townhalls, though they use softer language than I do. Since 2017 a lot of modernization attempts have been made(go cloud, use standards, use off the shelf software as much as possible), with very little to show for it so far.

Obviously it isn't all just tech that causes this, culture has a big part to do with it.

It feels like the scenario in the phoenix project almost, it'd be funny if it wasn't so serious.

Resonates with some big corp's I've worked with in the past; especially in industries where technology (rather than old business models) are becoming the primary channel of sales. Technology companies with technology management often disrupt the companies managed by the old way of thinking (e.g. finance, insurance, etc).

Some of the things you mention are red flags though at least to me having worked in them before - for me they normally make me question the companies management. The biggest one seen in a previous place I worked IMO - is buy vs build as much as possible off the shelf. How many successful tech first companies who have disrupted actually use that model for their core platform? The companies I've seen get away with it until they face competitive pressure OR technology isn't their primary advantage. In fact as you get to a certain size it can make sense to build your own and reduce your vendor count + ongoing costs and take advantage of your economies of scale. How many big tech firms rewrite db's, parts, dev-ops tools or are at least open to when the advantage is there? The successful tech first companies often do, even if they were once things like book stores where it "wasn't their core business". They even open source their components often to support their business and give their tech people more cred; allowing them to attract even more tech talent. The most successful/nice to work tech places usually err to building when it concerns their platform, with some pragmatism thrown in to use modern tooling from elsewhere if required normally open source but can be bought if it offers nothing of differentiation (e.g. cloud products, databases, etc).

Cloud is just a potential enabler IMO; you still need the culture to execute. A big corporation has a lot of interacting requirements, and needs the long term flexibility to change it without being on the hook in a vendor's backlog competing with other firms. It also potentially leaks your roadmap to other competitors. Common business software (e.g. document writing, email, chat, etc) are the exception; if your a big corp your usually in a monopoly/oligopoly position - there aren't too many people doing what you do at the scale you are and most vendor solutions are really just "outsourced builds"; where the long term flexibility as the vendor pivots/changes deteriorates.

That is definitely not technical debt.

Technical debt is the idea that just like you'd take a loan for your business to grow faster, you can take shortcuts while coding your first versions (for eg. not having adequate amount of test coverage, not making it modular or extendible etc.) to get the product out or meet a deadline. Just like your business loan, you get into technical debt knowing that you will eventually pay it back (i.e. write additional tests, refactor etc.).

Not every software problem is technical debt.

While your problem seems much larger, the same problem happens in smaller scales in every software company when engineers quit and leave a complicated codebase behind for new hires to take over. You should be looking specifically for people who have experience working with legacy code in your next hiring round.

I don't know that it has any actionable information for you (or anyone, really...), but what you describe sounds a lot like the situation in The Phoenix Project.

That is definitely not lost on me. Life imitates art.

Reply on news.ycombinator.com