DSpinellis

GDPR applies to persons and businesses. The fact that you're violating GDPR, because without your knowledge Apple has deployed software on your computer that compiles this knowledge graph makes Apple complicit. Imagine violating a speed limit, because your car's speed indicator has a software flaw that causes it to show a lower speed.

My understanding is that collecting these data and constructing the knowledge graph violates the European General Data Protection Directive (GDPR). According to the GDPR, personal data (such as an email's sender name and email address) are provided for a specific purpose (in the case of sending an email, for identifying the sender and facilitating email replies). Similarly, a calendar invitation is provided to organize a meeting. Processing that data for a different purpose (as described here, for constructing a knowledge graph) without the subject's explicit consent is unlawful.

Same here. I used to hit back LLMs with refactoring ideas and prompts to use better APIs or reduce duplication / increase abstraction. Ever more they're doing such things without being asked.

We stopped learning how to calculate the square root with pencil and paper (I learned that; never used it).

There has been a dramatic shift in the role of agriculture in the US economy. As a share of it, in 1820 72% of workers were in agriculture, in 2017 this had shrunk to 1.6%. Tractors played a big role in this transition.

Given that AI-generated code is not 100% perfect, we need to take into account the time and cost of reviewing when we discuss the productivity boost AI developers receive from code.

I convinced a software company to use a version control system (RCS on shared disk) back in 1993. To make it work we had to setup a network — Ethernet over (thin) coaxial cable at the time. This was so new to us that we didn't know we needed to use terminators on the two cable ends.

Indeed! The repo includes some v4 elements: https://github.com/dspinellis/unix-history-repo/tree/Researc...

The provided kernel predates the actual edition by a few months. It is based on https://www.tuhs.org/Archive/Distributions/Research/Dennis_v..., which matches V4 more than V3.

I don't think the concept of a rebase existed before Bitbucker and Git.

30 years ago (1995) open source offerings: mostly CVS for large projects and RCS for smaller ones. On the proprietary side, the aged SCCS was available and used, while Perforce and Microsoft Visual Source Safe were being launched.

I published an updated extension of this post's linked article in Empirical Software Engineering. You can read it without a paywall at https://rdcu.be/b7FzE. You may also be interested to see the actual GitHub repository at https://github.com/dspinellis/unix-history-repo.

That's the point: non-dependant tasks can run concurrently in Airflow. In sh/BAsh/dgsh dependant tasks can also run concurrently, as in tar cf - . | xz.

A nicer syntax and a lower probability of deadlocks.

And, more importantly, assign a name to a process, so that it can appear multiple times in the graph.

Admiring your multi-tasking!

I'm curious: what do you mean by "dgsh will use iteration under the hood too"? Dgsh does several things under the hood, but I wouldn't characterize any of them as iteration.

Manually playing around with fds is definitely unmaintainable. My hope is that a clean syntax can help create maintainable complex pipelines.

Thank you for the suggestion. This would mean that you'd also then create some mapping from each name (like git_log) to its implementation, right?

Nice benchmark! This is a (not at all efficient) awk one-liner.

awk -F\; ' $2 > max[$1] { max[$1] = $2 } !($1 in min) || $2 < min[$1] { min[$1] = $2 } { sum[$1] += $2; count[$1]++} END { for (n in sum) printf("%s=%.1f/%.1f/%.1f, ", n, min[n], sum[n] / count[n], max[n])}'

Can't see how dgsh could be applied to it.

I've found creating pipelines with Python to be messy and intuitive. Other than creating a DSL to express them I can't see how DAGs can be expressed naturally with Python's syntax.

Even creating tools in Python that can be connected together in a Unix shell pipeline isn't trivial. By default if a downstream program stops processing Python's output you get an unsightly broken pipe exception, so you need to execute signal.signal(signal.SIGPIPE, signal.SIG_DFL) to avoid this.

I went through two iterations before adopting the current syntax. Truth is neither me nor Doug McIlroy, the inventor of Unix pipes, who kindly and generously provided feedback during dgsh's development, had something better to propose.

What syntax would you propose?

Author of dgsh here. This is definitely not what I had in mind.

Apache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.

Fifty years ago Marc Rochkind wrote SCCS, the first version control system, which later inspired RCS, CVS, and Git. In the March issue of the IEEE Transactions on Software Engineering he provides a retrospective of SCCS: its innovations, its problems, and how subsequent systems improved on it.

I advocate the following rules for when to write and when not to write a shell script.

# Write a shell script:

* Heavy lifting done by powerful tool (sort, grep, curl, git, sed, find, …)

* Script will glue diverse tools

* Workflow resembles a pipeline

* Steps can be interactively developed as shell commands

* Portability

* Avoid dependency hell

* One-off job

# Avoid shell scripting:

* Difficult to see the preceding patterns

* Hot loops

* Complex arithmetic / data structures / parameters / error handling

* Mostly binary data

* Large code body (> 500 LoC)

* Need a user interface

A need for associative arrays (implemented in Bash as via hashmaps) moves the task to the second category (avoid shell scripting).

Pipe-related concepts in various restricted forms were floating around for years. Doug McIlroy indeed proposed them in 1964 and was heading the Bell Labs team when they were implemented in the Third Research Edition of Unix (1973).

See https://youtu.be/FbDebSinSQo?si=xbWIqES80hI2S802&t=901

ChatGPT can generate wrong code but it can also be a great helper. I've summarized the good, and bad, and the ugly in a recently published IEEE Software column: https://doi.org/10.1109/MS.2024.3363848

The ai-cli library attaches to programs that offer interactive command-line editing and modifies their interface so you can obtain generative AI suggestions with a single keystroke.

A repository with GitHub Markdown and troff-inspired extended typography hotstrings for the AutoHotkey program. Input thousands of math symbols and emojis in your editor, chat, CLI, IDE, email, browser. ∄ a,b,c,n∈ℕ∧n>2 ∋ aⁿ+bⁿ=cⁿ 2H₂O + 2e⁻ → H₂ + 2OH⁻ I +

There is a series of conferences MSR — Mining Software Repositories — with research papers looking at such questions. http://www.msrconf.org/ In fact, I presented this work in the 2015 MSR conference.