Runme – Interactive Runbooks Built with Markdown

Runbooks are about eliminating toil incrementally.

If you or your team find there's some process that needs to be done regularly, you're right, it should be automated. But there are often barriers to doing that immediately, and you may not even agree on what all the steps are. Here's how you do it then:

1. Write the runbook.

2. Execute the runbook multiple times.

3. Modify and improve the runbook.

4. Automate parts of the runbook.

5. When the entire thing is automated, hook it up to a metric to run automatically.

6. Delete your manual runbook - keep this new piece of code documented with its history and steps of improvement and eventual automation.

This method means you get consistency of a method almost immediately, don't need to fight product to spend a ton of time on it (you can spend 30 minutes here and there on a single step), the team has better conversations about ad-hoc process, and you eventually automate it.

In my job we have to do code releases to customers. This means we take a repo, do some tidying up like removing internal comments, inline some dependencies, make sure the README is in good shape, then make sure several people are notified and can approve it.

I wrote a scrappy runbook with like 10 bullet points because this was being done really inconsistently. A month later it had been improved a ton by engineers with detailed command lines and roll back steps. Now they're starting to automate parts of it. We don't have explicit time for this, but we got a consistent process straight off the bat, engineer buy-in to the process and improving it, and bit by bit we will get to a point where it becomes a one-liner without us feeling we had to carve out a chunk of time specially for it.

What's not to like?

> Interactive Runbooks exist in some sort of tech-debt limbo where you have enough time to write a nice little runbook but apparently not enough time to properly automate whatever the task is

I think this is a really common place to fall into. Automating 100% of conditions is hard, but having a google-doc full of bash commands is kinda meh.

One of my previous teams decided on a "Maintainer Lambda" which would help with some tasks we couldn't automate fully. The Lambda could either make an action (trigger a db backup, trigger a bounce, etc) or it could perform some query (find FQDN hostname of server where ${INPUT} metric is in alarm). It'd help with calling complex APIs (eg. AWS APIs) repeatably, and testable, and it'd handle any permissions and input validations. The Lambda had unit testing, and it could run automatically under certain limited conditions. Over time, we've added to it, and let it handle more tasks, as long as it'd log it's actions in a ticket (eg. drain server, then auto-bounce on mem-full alarm).

We'd embed cURL scripts into the runback doc, to trigger the functions manually. But the biggest win was whenever we got an automated ticket, for things like alarming metrics, it'd auto-run and parse the ticket, spitting out useful commands into the comments that the on-call could run (eg. copy+paste'able SSH-to-host commands based on which server triggered alarm). It's basically a reverse-run book. Instead of looking up what to do, we let the ticket tell us what to do, then slowly automate those steps.

I have a bunch of things I want to use them for. amongst them

* tutorials and other show and tell type stuff.

* troubleshooting: have a collaboratively-authored runbook that queries APIs, loads/aggregates data in some way and allows people to interactively examine response data (like a debug terminal sort of thing but with graphing etc)

I think they're really exciting.

Wouldn't Ansible be the better runbook then?

I haven't played with any of the alternative/independent implementations of Org mode but there are several out there¹. It would be cool to use one as the basis for a tool like Runme, or a collection of plugins for other editors.

--

1: https://orgmode.org/worg/org-tools/index.html

Also, often you don't need to save your Jupyter notebook in the unergonomical json format; instead save it as a script containing your code cells (and the markdown cells as comments). Thus you can run it directly from the command line in whatever language your notebook is, python, bash... To save notebooks as plain text files you pip install jupytext, and it magically works: jupyter will interpret your .py files as notebooks, and so on.

That's how GitLab thinks about that problem, too: https://docs.gitlab.com/ee/user/project/clusters/runbooks/#e...

> The JupyterHub app offered via the GitLab Kubernetes integration now ships with Nurtch’s Rubix library, providing a simple way to create DevOps runbooks

although this is another case of the "don't use the word 'simple' in your description unless it actually is"

Very cool, thanks for calling that out, we are big fans of Jupyter!

This is a good solution

Join us, come hack on Runme - it's OSS and we are always looking for more contributors: https://github.com/stateful/runme

Haha now that you mention it, sure does feel like it. I think the music plays a big part as to why

We've seen that it's not unusual in operations for folks to walk through runbooks running commands that require small (but important) amounts of human input. Runme makes that experience considerably better by utilizing the markdown that's already there and allowing for behavioral configuration (user prompts, process execution, language interpreter etc). But you can also name, and run groups of cells/commands to streamline your workflows.

Yes, most likely. Many organisations are still tied to manual/semi-manual deployments due to company policy or regulatory compliance (banks for example).

I wouldn't squirrel my documentation away in an ipynb file. So even though I'd like to use notebooks as runbooks, Jupyter just isn't the right solution for that. This could be a great tool for when you want to augment markdown documentation with runnable snippets. I really like this idea, it gives off literate programming[0] vibes.

[0]: http://literateprogramming.com/

There are actually many differences. We focus more on operational tasks, and allow cells to contain code from a number of programming languages in addition to python. In fact most folks are using Runme to make executing runbooks full of bash commands a better experience.