Hacker News Reader

Comment by orwin | Hacker News Reader

If you have four of five sklearn tiles that take 20 minute to an hour to execute, you might not want to do that too often.

I did not use jupiter notebook/jupyterLab much, but each time, it was in the context of datascience. The first was on an OCR during my internship, the second for data exploration (mix of quantitative/qualitative, but the project was scrapped after a week or two). In both case, having to re-run all each time the kernel was shut down was actually a pain point.

View on HN

I encountered this a lot in my previous work and found workarounds that I write about here: https://rachitsingh.com/collaborating-jupyter/. At a high level I think being scared of rerunning your kernel is indicative of a code smell, and there's relatively easy ways to combat that.

Personally I don't like the "write to disk" approach; I think it kind of just punts the state problem somewhere else (i.e. from memory to disk). Writing to a database and adding versioning is better, but that's a lot of machinery to expect a notebook user to adopt (though maybe better tooling could help). Also a lot of Python objects are not out-of-the-box pickleable (e.g. generators). Also pickle is a mess.

I definitely agree (and I think given what you work on you would be horrified by how I define cached functions that capture locals), but I think in practice getting to a state where you can restart your kernel often makes it easier to reason about state. But you’re definitely right, it would be better to reason correctly here.

One thing I’ve toyed with is writing a Jupyter kernel extension that notes what new locals you’ve defined in a cell, figures out what locals are read, and creates a (cached) function from the cell. E.g. a cell that has `y = a @ x + b` becomes

    @cache_to_disk
    def compute_y(a, x, b):
        return a @ x + b
    y = compute_y(a, x, b)

I don’t worry much about serialization - 90% of the time what I need to cache is dataframes (write to parquet), and the rest is trained models (custom serializer). People rarely need to cache generators, in my opinion.

Yeah, I was definitely a python newb, barely used generators or context managers.

I ran them once every morning still.

Yes indeed - not having to re-run expensive calculations is one of the major appeals of the notebooks. But still, I believe it should be done reasonably often - at least once, before committing for example (if you're using tools like jupytext to convert notebooks to .py)

Reply on news.ycombinator.com