JupyterLab 3.0

can look into google colab to directly load the notebook. Link : https://colab.research.google.com/

Yeah I've had this issue in the past with Jupyter. Now I generally look for docker images when I want to check out something new.

True. I like Python, but the ecosystem can be very confusing for a quick dive. It's easy to get lost between pyenv, virtualenv, pipenv, pip, pip3, easy_install and friends.

Not sure about Mac, but on Linux, you install some docker images and it just works (modulo some docker flags to learn if you don't already know docker)

I have used Python on OSX for years and it is and always will be a horrorshow. Using the system Python installation is a nonstarter for many reasons, chief among them is that I don't have any interest in using py2. So then you're using pyenv or homebrew, but your vim install still thinks that it should be using the system python. And whoops, you fixed that and now virtualenv is not finding your interpreter. And etc., etc.

OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment.

As always xkcd has a comic for it: https://xkcd.com/1987/

Relevant XKCD.

https://xkcd.com/1987/

You should use Anaconda instead of pip to install data science related tools in Python.

```

$ conda create -n {new_environment_name} python

$ conda activate {new environment name}

$ conda install jupyterlab

```

Always try using a virtual env. Also try using pip directly through ‘python3 -m pip install jupyter’ . Same to run it to bypass path issues ‘python3 -m jupyter notebook’. You should be up and running in a couple mins!

Yeah, I use portainer and then load a docker image of Jupiter and import the notebook there.

Dependency management hell isn't just a Javascript problem these days.

Just install this: https://www.anaconda.com/products/individual

Not sure about the Jupyter-specific issues but macOS comes with Python 2 installed, so you have to install Python separately and use the `python3` binary, or alias `python` to the `python3` binary.

Then remember to run pip using `python3 -m pip <command>` and use virtual environments (venv)

I stumbled on this a couple weeks ago and documented the process here https://flaviocopes.com/python-installation-macos/

I found the same problem with Ruby (old version preinstalled on macOS)

> I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie!

Python is a little worse in that in pushes its dirtiness downstream to your users too.

Python is AMAZING and Awesome TILL You want to spread what you are doing to more then one computer. Though I have to say I love domain specific languages and personal enjoy Racket as my most fun language to use. It is sad that this has gotten so much worse over the years.

In a better world, the notebook you downloaded should have come with a list of dependencies and have reproducible code in it. That said, Python environment tools and package managers are still in a state of flux. Some tools make it easier, but are not helping much with reproducibility (plain venv + pip, but only if you got all required system packages installed!), other tools enable reproducibility, but have growing pains and are more difficult to use or get working (pipenv, poetry).

If you are in luck, your required libraries and appropriate versions are available as GNU Guix packages and you can use that.

No problems on Linux. I use pyenv for managing python versions and virtualenvs.

At the risk of contributing my own probably-subtly-broken advice to the collection already present in this thread, I would note that there is a relatively sane system python3 on Catalina/Big Sur, and most projects are building and distributing compatible wheels [1] (binary packages). So something like:

    /usr/bin/python3 -m venv ./venv # create a virtual environment in ./venv
    ./venv/bin/pip install wheel # optional
    ./venv/bin/pip install dep1 dep2 dep3 ...
    ./venv/bin/python ... # up and running

is enough to get started in most cases. Of course, all bets are off for GPU or Apple Silicon support...

[1]: https://pythonwheels.com/

>I don't think it's Python's fault.

Actually it is:

https://xkcd.com/1987/

That said, you would be up and running if you used conda.

And this is not the usual "just try distro X" Linux advocate BS when somebody has an issue with distro Y.

Conda does makes thing more user friendly by focusing on whole environments. Actually that's why it was built for.

Note that this applies for this situation. For other uses cases, e.g. deploying something to production, having used conda would have made your life more difficult.

virtualenv makes things a breeze

Funny. I wanted to try this so I just ran the docker image and it worked. Time spent: 2 mins.

docker run -p 8888:8888 jupyter/scipy-notebook

I'm in the same boat. That's why I use repl.it[0] for nearly all my programming now. Haven't tried jupyter on repl.it, but here's[1] the first DDG result, which suggests jupyter is supported.

[0] https://repl.it/ [1] https://repl.it/talk/templates/Jupyter-Notebook/49993

I find that everything in Jupyter-land is as inconvenient to reproduce as it is convenient to experiment with. Case in point: last I checked, you still had to do gymnastics in order to even store your notebooks in Git in a meaningful way (e.g. allowing merges).

It really throws me off the whole thing (Julia + Jupyter is even worse...) because I’m not keen on putting effort into stuff that nobody, myself included, will be able to use in a year or two.

> I don't think it's Python's fault.

Well, there's a long legacy of half-baked solutions and kludges. And, especially, install instructions consistently suggest you do the worst thing: install stuff using pip.

Everyone knows that's bad advice, because it always fails in mysterious ways, and there's no good way to roll everything back.

I recommend:

    python3 -m pip install pipx  # Don't rely on the pip3 bin.
    pipx install jupyterlab
    # Now jupyter has an independent virtualenv.
    # To install dependencies:
    pipx inject jupyterlab numpy
    # Where is everything?
    pipx list
    venvs are in /Users/ben/.local/pipx/venvs
    apps are exposed on your $PATH at /Users/ben/.local/bin
    ...

Once you've done that, never install anything directly with pip3 again. If you're working on a project, use pipx to install poetry / dephell / pipenv and use them to manage the virtualenv.

And you can get rid of it all:

    pipx uninstall-all
    python3 -m pip uninstall pipx

My most recent professional project involves - among other things - a Python API deployed to AKS. I'd used Python here and there for fun and on the periphery of work, but this was the first time I had to dive into it full-on.

While I know it has earned its praise for good reason -- as I'm sure all the data scientists I work with would attest to -- as an engineer, Python has been really frustrating to work with.

Don't even try. Use docker, there are official pre-built images. It's easy:

  docker run -p 8888:8888 -v ~/Code/actual-project/:/home/jovyan jupyter/datascience-notebook

Note ~/Code/actual-project/ is a git repo or a mounted Google Drive folder. Moving from one machine to an other is pain-free.

[1] https://jupyter-docker-stacks.readthedocs.io/en/latest/using...

When following the docs, Go just works. It compiles fast, though not so dynamic as Ruby and Python. Tooling and runtime also matters in one's work.

Had same experience.

Hoping to help clear some backlog, I tried to help with some ML & data pipeline house cleaning. Had never used Python before, but how hard could it be? OMG. One really doesn't appreciate ease and simplicity of multiple simultaneous JDK installs, all peacefully coexisting, until working on some python and nodejs projects.

Just running user space dev env via Docker was recently front paged here on HN. I believe that will become the norm.

Being a macOS user, I looked to see if there was something more lightweight. Like maybe a port of podman. Lighter than running a virtual Linux instance.

Again as a macOS user, I feel like I'm overdue to embrace the remote dev desktop set up. I've read that Googlers don't do any work locally; they all remote shell into servers running the tool stack. So very X Windows; what is old is new again.

That was an explicit decision in the underlying UI toolkit, which supports overriding the context menu in a generic way, to move the browser's default context menu to Shift-Click.

Ugh, that's a lot simpler than my solution. I found out (I think through SO) that if you select the 'Create New View for Output' option, you can copy and paste as normal.

I'll have to try that next time I'm struggling with pandas.

One thing to look into is figuring out some logic of cancellation to clean up the generated code. Even if you just checked back one step it would mean less Foo_Bar['D'] = 0.

I think saving the analyses in the home directory is defeating the purpose. One appeal of a notebook is that you can break all the steps out and show them to someone, add commentary, etc.

I'd want to use it interactively to figure out how to get my data looking the way I want, and then show off how the analysis works by pulling out non-interactive Step widgets that visualize changes into their own cells.

That said, I'm often doing stuff with numpy so I need the notebook to document exactly what the dimensions mean, why I did a "clever" thing, etc.

Also, as long as "import mitosheet" is hitting your analytics APIs, it's going to set off alarm bells if we try to use it inside our VPN. The compliance people are not likely to look at what it's sending or why, they'll just ban it and move on.

Looks nice! Which spreadsheet/grid library have you used, as I've not found one performant enough once you get 1000+ rows?

It’s not a basis for their products but this is something https://colab.research.google.com/notebooks/intro.ipynb#rece...

Maybe I misunderstand something, but I use jupyter for sage for example and the point of it is that graphical results can be shown there immediately. Some plots can be even made interactive.

> why do people continue to do their work in one? Most nb users don't use vscode. I'd say main reason is you can't open a notebook without starting a kernel which takes >1 second. It also is harder to setup.

In terms of a flat file formats, there is actually a mark down extension which I would say is better than vscode's format because it is more standardised:

https://myst-nb.readthedocs.io/en/latest/

I don't think anyone who writes production code is using Jupyter as a development env. It's more like a self-documenting, persistent repl, or perhaps an interactive doc.

I've used it as a way to document data exploration for later use and as an interactive teaching tool. For that, being able to plot inline and embed interactive widgets is great. Also, mixing in rendered markdown content.

If you're using Jupyter to write python software and the software is the end result and not the data analysis/process, I think you might be doing it wrong.

Not that I'm discounting data scientists (or the like) who use it as their primary environment. Their end result is not a packaged/deployed software and the code itself is just ancillary. For those users, the code and environment is a tool, and the extra weight of what a full IDE (like VSCode or PyCharm) and the features they bring to the table are unnecessary and likely counter-productive.

I think you're suggesting that if you want to retain both code and results you need to manage two different files - raw python and exported notebook. I think many notebook users would see that as inconvenient (and many are not familiar with VSCode).

> when there are editors like VSCode that can treat ordinary python files as notebooks

was doing this in emacs before there was a jupyter notebook..

as the name implies, notebooks aren't for development, they are for interactive, shareable presentation, and presentation that also works over HTTP without needing local tooling

I use PyCharm + Jupyter Notebooks with autoreload. I'll build up functions in the notebook by running small bits of code, then move them to a Python file. It's like a supercharged version of a REPL.

VSCode's version is nice for sure, but not nearly as nice as being able to go back and read/modify old cells.

I've seen it crop up here and there. I've used it a few times and its really amazing how fast.

Afaik it's a drop in replacement for conda using the same cli parse. Even has miniconda (micromamba).

Our solution to this is just to publish notebooks to an internal gitlab repo. Easy to share read-only versions, it renders a bit different but looks close enough. You can then easily manage user access rights in gitlab and contributing back changes is easy with the jupyterlab git integration.

For publishing interactive examples we use voila, which creates a dashboard version of the notebook for users.

jupyterhub addresses some of these concerns.

Mamba is a reimplementation of the conda package manager in C++. (quote from README at https://github.com/mamba-org/mamba, linked to in original post)

Have you looked at any of the Jupyter Docker Stacks images? [1]

I've built some personal Docker images based on these with quite a lot of additional features installed including Ruby, OpenCL and lots of additional Python and R packages and JupyterLab extensions. It's been a bit hit and miss - with extension incompatibilities being a bit of an issue. I sync directories on the host machine with the Docker image and so have the benefit of editing in a full text editor (for non-notebook scripts).

I'm not a Python env expert so I've probably missed a few tricks but all seems to be working well now. Happy to help if you want to try this route and have any issues.

Btw after using JupyterLab for over a year now I'm a huge fan.

[1] https://jupyter-docker-stacks.readthedocs.io/en/latest/

I use a setup like this. Essentially, I use pyenv [1] to manage Python versions and Poetry [2] [2](https://python-poetry.org/) for virtualenvs/dependencies.

The workflow for creating a new project looks like this:

1. Create a project directory (e.g. 'myproject') and `cd` into it. 2. `git init` 3. Fixate the Python version for that project with the `pyenv local` command (e.g. `pyenv local 3.8.6`). This creates a `.python-version` file that you can put under source control. Within the `myproject` directory tree, `python` will now be automatically resolved to the specified version. Your system Python (in fact, any other Python versions you might have installed) remain untouched. 4. Create a new poetry project (`poetry init`). This creates a `pyproject.toml` which contains project metadata + dependencies and can also be checked into git. 5. Add dependencies with `poetry add`. Here, you could for instance add Jupyter Lab (`poetry add jupyterlab`).

To access installed dependencies, such as the `jupyter lab` command, you can either execute one command in the virtualenv directly (`poetry run jupyter lab`) or spawn a shell (`poetry shell`). If you open a Jupyter Notebook that way, the packages installed in the virtualenv are directly available from within Jupyter Notebooks, without having to mess around with installing IPython kernels.

I like this approach, because it gives you full flexibility, while being portable and easy to use. It gets you around having to deal with conda (which I found to be frustrating at times). Also, you're not tied to the Jupyter frontends, but could e.g. just install `ipykernel` and open notebooks in VSCode.

[1](https://github.com/pyenv/pyenv/) [2](https://python-poetry.org/)

Edit: Moved the links

I have an unconventional setup of Jupyterlab, Python dependencies, Swift (through PythonKit package) with Bazel. Surprisingly, the new `rules_python` and `pip_install` support works great at installing packages through pip. This ensures on any machine, I will have a consistent Python runtime (either downloaded or build from the source), as well as the pinned python dependencies when the repo checked out. It also helps because I have a Swift kernel that packaged inside the repo as well.

One thing I haven't figured out is about Jupyterlab's extension system, which previously requires node.js for delivery. It seems 3.0 removed that requirement, so I am hopeful with some tuning I can deliver the plugins in consistent way as well.

On linux, I install Anaconda and just launch JupyterLab from Anaconda Navigator.

But you could just install conda and just `conda env export -n base` to export your environment.

mamba is (nominally) a drop-in replacement for Conda's CLI and dependency resolver. I've been using it for a couple of months and it really does run an order of magnitude faster for anything that requires dependency resolution. You'll still want to keep ana/miniconda around, but it goes to show how some of Conda's performance woes are entirely self-inflicted.

Could you share any examples of what you've seen?

Are you looking for PhosphorJS? https://phosphorjs.github.io/

It's the UI platform Jupyter built to power JupyterLab but is also a general purpose UI framework.

I do. The find and replace is better. You can do it across multiple cells. I don't like the drag and drop method of moving cells. Switching between notebooks is faster if you normally use multiple large notebooks at the same time. I am biased against JavaScript heavy UI. I just have had so many bad experiences with single page apps. The only problem I have with notebooks is the mathjax J's seems to cause the browser to hang on opening or running large notebooks. But it was the same in labs, but even worse with the tabs being handled by the UI instead of the browser. Switching back and forth between tabs and having to wait for them to render was annoying. I gave it a good go for a while recently, but the find and replace in classic notebooks is just so good I couldn't make the switch permanent.

Yes! Every now and then I give JupyterLab a go but I keep going back to classic notebook. The only two extensions I need is Vim key bindings and Black formatting. I might checkout version 3 in a few months, but classic has a “feel” that just works for me. I’ve also tried the VSCode abs PyCharm notebooks. The dealbreaker is always how it displays the output of cells. It sucks compared to classic.

Yes, debugging notebooks is not the best experience so I'm glad there are improvements. I will have a look.