Hacker News Reader

Comment by shakna | Hacker News Reader

I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.

Reality is a messy thing.

View on HN

Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.

In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?

Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.

We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.

In cases like these, I'm a strong proponent of turning it off and seeing who yells.

In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.

There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"

future reference: next time just firewall the network and see what blows up rofl.

A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.

The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.

Reply on news.ycombinator.com