Comment by Too | Hacker News Reader

Comment by Too | Hacker News Reader

Automate everything that can be automated. Avoid setting up things using GUIs.

Starting a set of services should be as simple as "docker-compose up", building should be as simple as "make", checking out the code should be as simple as "git clone", etc. You shouldn't need a shitload of wiki checklists that describe how to install dependencies and how to check out all the git-directories with correct versions relative to each other. Save your wiki for high level documentation

A must for this to work is to avoid configuration state in your servers that is not captured in your source tree. This is the most critical because it's easy to forget and it can quickly become a blocker, not just for RnD but also ops. You have this super important service that everybody depends on and it always works, suddenly the server burns while go-to-guy is on vacation and nobody knows how to bring it up again from a clean slate because it involves several hours of clicking around in some proprietary GUI and ticking all the correct boxes. If you can't reproduce this service today, make a VM snapshot already, then start work on fully declarative service configuration.

That's good technical advice but I don't think it works as far as knowledge capture goes. There are two problems.

Firstly, it fails to solve the problem of actually capturing knowledge. In fact, if anything, you're suggesting that knowledge of the systems and processes shouldn't be necessary in order for the business to function and that building a black box that "just works" is good enough. The problem with that is two-fold. First, using code to capture knowledge (eg "read the makefile to see how it works") fails to capture any reasoning for decisions that have been made, and secondly any history of the changes to the system are lost if you do ever decide to wipe out the git history of the repo (eg a shallow clone, or a squash, etc). Those may or may not be important to you but I've found it useful in the past.

Secondly, there are non-functional requirements for things that can't really be captured in code. For example, "The system relies on an external service that can only be rebooted by calling 555-1234" is knowledge that no amount of single start up command automation can fix if there's a problem. That needs to be in a recovery policy document so everyone knows where to look if the system fails. That way the document can be reviewed by non-technical people as well which is a huge bonus.

I've just had to script an earlier manual process and the last couple of steps couldn't be handled in code. The final log message, written to the console, says, "TODO: Now do this thing ...". So everything is under source control and there's a golden source of truth.

Imagine that a guy who automated a certain part of the process just left, and you need to make some changes. Who's holding the knowledge about how X is automated and the thought process behind it?

Ideally, the automation is available and can be read through to divine the steps it is taking to perform the task. As for the thought process, pray that the author left behind hints whether that be in the form of comments in code, descriptive commit messages, etc. Otherwise it gets a bit more tricky.

Comments and commit messages are documentation and knowledge capture, but done in a way that's really hard to read through and requires technical knowledge and repo permissions to even access. If your team includes people who aren't developers and you need to review a process it's very useful to have that knowledge in a more readable format.

I don't disagree with the overall point, but I'm not sure a non-developer would have a need to understand the specifics of how a task is automated. In theory, the automation is built off a defined process already which should be readily available to non-developers already.

In practice... Well, the initial question was about good practices so perhaps we shouldn't lift the lid on bitter experience.

I don’t like having extra steps to convert things to more readable formats because it’s expensive and boring and tends to get out of sync very quickly.

If non-tech people on a team need to understand, in detail, a technical process, I think it’s easier to teach them to read through a script than to pay a tech writer to document the script, all the time.

I’ve run into this problem with non-technical project managers who want to understand in detail. If this level of understanding is necessary then it’s possible to scan a source file to read comments.

Or generate documentation from source and dump it on a web server somewhere.

Although lately even non-technical people are able to read and edit markdown files on GitHub.

Yeah, I think if you are looking at commit messages to understand a process, you have to take a look at if/where things went wrong.

At worst, the code is there. This is much better than a person leaving who did some stuff manually.

Obviously additional documentation would be great, but code itself is a minimal type of documentation.

Yes, but the lack of reasoning behind the implementation means that the new person has to reverse engineer it (practically speaking). This isn't always bad, but in most cases it is.

My point was that by simply automating the process you don't institutionalize the knowledge as much as you move at least part of the problem somewhere else.

It heavily depends on the application, though. In my domain the implementation is rarely self-documenting, and most of the system's complexity is usually baked into the spec or the human part of the process. In some other domains the proportion is completely different.

This is great until you need to change the automation, or something it depends on breaks. Hiding it from a new employee is a wise idea, however, but at some point they may need to know these things and for that to happen it needs to be, you guessed it, documented.

Reply on news.ycombinator.com