Hacker News Reader

Comment by ed_elliott_asc | Hacker News Reader

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares?

Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong.

It comes with a well documented security mode and a support contract when needed.

Developers can be onboarded quickly and work code reviewed and managed.

Logging and diagnostics are available and you can report on metrics easily.

That isn’t true with custom data pipelines written in shell scripts.

The value isn’t in the pure execution time, it is in everything around it.

View on HN

Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale.

It's the same narrative people use for many complexity or slow things

> The value isn’t in the pure execution time, it is in everything around it.

Especially for those of us who get paid for developing, designing, supporting, architecturing, meetinging, catering, conferencing and maintaining everything around it :-)

> That isn’t true with custom data pipelines written in shell scripts.

Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.

This is like the classic Dropbox hackernews comment:

> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.

Sure, you can stitch several services together and it will work for your needs, but for most users there is a benefit to a centrally managed and complete solution.

> but for most users

Define "most users"

Most users who have to tackle actual big-data problems, meaning analysing things on the order of several TB or more?

Sure, they will absolutely benefit.

But there isn't just big data. There is also little data, where what is analysed is on the order of a few GiB or less, and everything in between.

I am not saying "use shell for everything!" I am saying "the right tool for the right job". A 15t excavator is probably not a good choice if I want to plant a small tree in my backyard, and a gardening shovel will probably not serve me well when I wanna start building a scyscraper.

Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands?

Or is that more of a kubernetes thing?

You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.

That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only.

https://github.com/dask/dask-labextension :

> This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes.

Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub

I don't see why this is downvoted. Seems like a solid argument, even though I don't agree.

Reply on news.ycombinator.com