I just taught a course to a client this week helping them with this (and other best practices for Python).
I just taught a course to a client this week helping them with this (and other best practices for Python).
Currently leaning towards no, as they are hard to test, compose, and version control (it is possible, but you need bespoke notebook-specific tools, and most just don't).
Any time a notebook needed to be tested, refactoring it into a module and testing that was the better choice.
Notebooks in my opinion are by far the best tool for interactive and exploratory data work. I've been using Jupyter/IPython for 10 years and it takes very little discipline to keep them clean and clear. I've never bothered trying to deploy them in any meaningful way.
The quirks with version control are annoying, that's one reason I've switchex to marimo in the last few months.
Do you have this written down anywhere or on video anywhere? I'd love to learn more about what you mean.
When you are exploring something, experimenting, showing.. it’s great; train-of-thought structure, APIs like Pandas optimised for writing and terseness etc.
But when you have a piece of code that will lose a million dollars a minute if someone ships a bug, and which will be maintained by many engineers over many years, then you really want a format that’s optimised for long-term maintenance, incremental change, testability, and APIs optimised for readers.
Personally, I think the fact that notebooks are usually easier/funner for me to work with is a big problem. I'm by no means a Clojure expert, but I did do a semi-large project in Clojure a few years ago, and some of the ideas of true REPL-driven development that exist there are things I wish that Python supported.
It's hard to explain without actually learning it for real (and most Python devs mistakenly think Python has REPL-driven development; I sure did before learning Clojure!). But once you get used to being able to interact with your actual source code, and at any point just being able to write new code and immediately print out its value, then with one shortcut make it part of the regular codebase... that just blurs the distinction that exists between Jupyter Notebooks and production code in a way that makes everything much better.
What tools are used for "write new code and immediately print out its value, then with one shortcut make it part of the regular codebase" and how does that square with working on a team and getting code reviewed?
Writing clean data code is one aspect. Filling in knowledge gaps is another. Covertly teaching software engineering best practices to folks who "aren't programmers" yet sit down and write code in Jupyter all day is another.
I should probably write a blog post or record a short video. (I just taught a week long course on this for a client last week.)