You'd rather have plain text than hand-written diagrams? I think most of the difficulty is in not knowing APL like notation.
The handwritten bound notebook cannot be searched, diff'ed, or versioned. It is strictly inferior to the plaintext file in most ways that are in any way relevant to writing software.
Its nemesis, of course, is and has always been the mathematical equation, whose notations confound the plaintext writer at every turn. For those, you can write it to an image, or use LaTeX or MathML.
It is just as important to control process documents as it is to control source code files.
(But I do use paper some - focus reasons, and easier to manage than terrible phone interfaces.)