Hacker News Reader

Comment by throwaway894345 | Hacker News Reader

> I said this already 10 years ago: We don't need more cloud computing but need to empower users end devices again. Jupyter is typically operated on powerful notebooks and not on mobile devices.

If you’re working with data of any significant size at all it then it doesn’t matter how fast your user device is—it’s so much cheaper (time and network egress costs) to send the computations from a user device to the cloud than to pull tens-thousands of GB of data to your local machine. Moreover, I don’t know of many local machines with tens of CPU cores, hundreds or thousands of GB of RAM, or tens-hundreds of TB of SSD for handling that computation quickly.

User devices are great for very small data, but I don’t see the point for larger datasets.

View on HN

Most users don't have 'tens-thousands' of GB of data as part of their use case. You're describing a business case, not an end user consumer case.

I don’t believe that the majority of Jupyterhub users are “consumers” rather than professionals, but more importantly, that doesn’t change the fact that the professional use case exist and isn’t amenable to the fat client approach.

As an anecdote, I work on a Jupyterhub managed service offering with customers in both the private and public sectors and our data sizes are pretty much all in this range.

> I work on a Jupyterhub managed service offering ...

That sounds like selection bias.

It might be. Like I said: anecdote.

Isn't it, though? The people most likely to need a cloud-managed service like that probably have too much data to crunch on a laptop, as you described.

Ah, to be clear I’m not saying our users are a representative sample of Jupyterhub users. I am saying that there are a lot of people who use Jupyterhub for large datasets—it’s certainly not uncommon.

Most of the users will not run Jupyter Notebooks locally, they expect a remote kernel in the cloud with GPU/multicore machines, only the web app will be running in browser (think of big query). Of course small users with you datasets will run locally.

This depends greatly what you want to do. After all, this has incredible latency (compared to the cloud), limited throughput. Zero configuration. Free.

You want to train the next imagenet model? Analyse 100Gig database? Probably not the correct tradeoff.

You want 20 students to have a perfectly consistent instant-start-up python environment? Definitely the correct tradeoff.

You want to try some python methods, write tests, ... Very short latency is going to help you more than throughput is.

To be fair, "we don't need more cloud computing" doesn't mean, "we don't need any cloud computing".

I don't agree that we don't need more of cloud, IMO we do, but we need to focus on personal computing much more than now, which is the general theme of the GP comment.

Not everything is big data.

To be clear, I’m not talking about big data. More impotantly though, I didn’t say there was no place for client compute, only that it isn’t economical for datasets in excess of a few GB.

Not only do cloud services offer better compute capabilities (and GPUs/TPUs etc), but they offer easier reproduciblity and sharing. Even when I hack on stuff myself, Colab is quick and easy to set up, no worrying about Docker or virtualenvs.

You definitely shouldn't be running this stuff from a Jupyter notebook.

Why not? We make a lot of money offering it to people.

Yes, the market in enabling people to do the "you shouldn't be doing this" stupid things is huge.

Any substance on why we shouldn’t be doing it or why it’s stupid? What’s the alternative? Should researchers all learn Kubernetes and AWS and deploy their own environments?

The problem with Jupyter is that it impedes common-sense practices like version control, reproducibility, and automation.

If you're spending the time and effort to rent these big servers, why not spend the 5 percent of the effort and do it right?

Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices. Must be some sort of us-vs-them point of misplaced pride.

> Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices

Bahahahaha, yup those dang lazy mathematicians just shooting themselves in the foot and forcing us to deal with it! /s

Your preferences for software development are irrelevant. The value is in delivering the math to the end user. Using GCP, kubernetes or JavaScript to do that is an implementation detail. Sorry to tell you this, but you're a servant to those dang lazy analysts and without their insights, you're worthless.

You misunderstood.

The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore.

Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.

I understood fine. The problem is trying to formalize "ad hoc" versus "production development" practices as if they're meaningful.

There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.

Your boss will eventually ask to make your "ad-hoc" stuff "production".

At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch.

If that's your thing, then go for it.

That’s fine. Why waste time productionizing something that may or may not ever go to production? We do this all the time. Do you think Tesla built a whole fully-automated factory to build its first proof-of-concept car?

> The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore.

Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other.

> Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.

This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.

>The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore.

Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.

I think that's a misguided take. I'm a software developer and data scientist (of sorts). Jupyter is an extremely convenient tool for adhoc data analysis. By default it gives you easy visualizations and inspection ability, allowing you to verify intermediate computation before rerunning.

It's easy to convert a notebook into a script with version control, reproducibility, and automation.

I am shocked at your unjustified and rather arrogant gatekeeping here.

Version controlled Jupyter notebooks running in an automated environment (eg Kubernetes), with repeatable test data loading into a processing environment (eg. MinIO or Spark), is quite commonplace. Making it even easier with WASM makes sense.

What about Jupyter impedes good practices? That it empowers ad hoc exploration at all? It is merely an IDE tailored to sharing interactive text and code. To me it is one of the most exciting ecosystems for modern software development (and I’ve been developing software for 30 years).

Reply on news.ycombinator.com