Uh, I don't know. This font is, well, artsy, but I find it honestly hard to read. It's just too unusual. I'm not a fan of some other decorations here as well (like these blurry raster pencils, for instance).

As for your question, usual answer would be LaTeX + (maybe) some specific plugins (like TikZ for diagrams, or tufte-latex for specific layout, or whatever), but due to some irregularities here I suspect that this particular book is actually hand-made in something like Microsoft Office. I well might be wrong, I'm not sure, it just seems like that.