drothlis

A lot of commenters suggesting "pipefail" aren't realising the full extent of the problem. Here's an example that might be clearer, from https://david.rothlis.net/shell-set-e

This prints “a”, as you’d expect:

    set -e
    myfun() { printf a; false; printf b; }
    myfun
    printf c

...because the “set -e” terminates the whole script immediately after running the “false” command.

But this script prints “a-b-True”, where some people (myself included) might have expected it to print “a-False”:

    set -e
    myfun() { printf a-; false; printf b-; }
    if myfun; then
        printf True
    else
        printf False
    fi

The “set -e” is ignored completely. Putting another “set -e” inside the definition of myfun doesn’t make any difference.

`|` is a bitwise-or in Python, not a boolean-or.

Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.

See also gittup, an experimental linux distro where the entire linux kernel + userspace is built using tup. https://gittup.org/gittup/

I did hit a weird edge case in Make's job scheduling. Ninja will try to build things in the order that they are listed in the "build.ninja" file; make kind-of does too, but if any dependencies need to be built first, the later targets get pushed to the very end of the job queue. This means that you can end up running many resource-intensive linking jobs all at the same time at the very end of the build: https://lists.gnu.org/archive/html/help-make/2016-11/msg0000...

It's hard to say how much impact this would have in real-world scenarios.

Yeah, meaningful benchmarks take a lot of work.

- Does the reported ninja time include the cmake step that generates the ninja files? (which typically doesn't need to be done for incremental builds).

- "make -j" doesn't limit the number of jobs, so you can run into resource contention if building too many things at once. Ninja (by default) limits the parallel jobs to the number of CPUs.

- Disabling make's built-in rules (with `-r -R`) can make a big difference to performance.

- Make spends a lot of time processing the ".d" files with header dependency information -- 98% of the time in my (artificial) benchmarks!

> So no include files? That's another deal-breaker!

Tup monitors any files that are read while the build command is running, and implicitly adds them as dependencies.

Tup has some automated performance tests, which I co-opted to benchmark Ninja vs. Make: https://david.rothlis.net/ninja-benchmark/

Do you mean VSCode’s standard git support, or is there a github plugin specifically? (Just curious if I’m missing out on something!)

That “cap” wasn’t a previous policy. It’s the amount of tickets already sold!

Just watched Daniele's talk. Brilliant, practical advice. Thanks so so much for the link.

Congratulations! Our own journey was fairly similar: Unsatisfied with existing tooling, created open-source project with blessing of employer (it helped that in <2 weeks we built a prototype that was already better than what $big_consultancy had spent months building), adopted by a few companies worldwide, eventually left employer and started an open-core business around it (albeit without VC money, and probably a much smaller market —pixel-based GUI testing for set-top-boxes—, and there's a hardware component to our product). Going strong since 2012 (open source project) / 2014 (business). I don't know that my advice will be relevant given the different market / VC approach, but feel free to reach out if you want to chat.

Many distros, including Debian & Ubuntu, have merged /bin and /usr/bin, with symlinks for backwards compatibility: /bin -> /usr/bin (and similarly for /usr/lib etc).

https://www.freedesktop.org/wiki/Software/systemd/TheCaseFor...

Mentioned in yesterday's discussion on declarative schema migrations for SQLite: https://news.ycombinator.com/item?id=31249823

I don't know about Django but in Rails† the migrations are numbered with a timestamp (in the filename itself), instead of a sequential number, precisely to avoid (git) merge conflicts. Then they're run in alphabetical order. There's a dedicated table to record which migrations have been run already.

† 10 years ago, anyway.

Ha! I use `textwrap.dedent` all the time but didn't make the leap of logic to check for the existence of `indent`. :-)

I suspect this was a leftover from Python 2.7 days.

We are big fans of Datasette (though I've yet to use it in real life). I think it's important work.

Thanks! Coincidentally one of the first articles I wrote for my blog was about compile-time checking of table/column names: https://david.rothlis.net/d/templates/

(Just something I was messing around with, not anything I ever used in production.)

> The PEG will parse the first a, and then the recursive call will parse from a|aaaa. Now the next recursive call will eventually fail.

Thanks for taking the time & effort to explain! I know you're right because I have reproduced this behaviour in 2 parser generators now (pegjs & grako).

Why does the next recursive call fail? (Don't answer, I think I've worked it out.)

Isn't it supposed to backtrack:

    1. a|a a a a
    2. a a|a a a
    3. a a a a a|  <- "succeeds" but doesn't leave any more input for the outer parses
    3. a a a|a a   <- doesn't it backtrack and try the other choice?
    2. a a a a|a
    1. a a a a a|

I guess it's because the backtracking doesn't happen at recursion level 3 (because it already thinks it succeeded) but at level 2:

    1. a|a a a a
    2. a a|a a a
    3. a a a a a|
    2. a a|a a a   <- tries the other choice
    1. a a a|a a

Let's see if my explanation holds for aaaaaaa:

    1. a|a a a a a a
    2. a a|a a a a a
    3. a a a|a a a a
    4. a a a a|a a a
    5. a a a a a a a|  <- "succeeds" but doesn't leave any more input for the outer parses
    4. a a a a|a a a   <- tries the other choice
    3. a a a a a|a a
    2. a a a a a a|a
    1. a a a a a a a|

Geez.

Well, now I'm grateful I've never had to implement a parser for anything more complicated than a Lucene-style query syntax.

I can't wrap my head around it. Why doesn't it work?

It parses this: a a aaa a a But not this: a aaa a

> Is there a mechanism that works well for improving errors in PEGs (i.e. something like a non-returnable node)

Yes, it's called the "cut operator".

> how does one practically implement that?

Pick a parser generator that supports it natively. ;-) If you're talking about implementing the parser generator yourself then you probably already know more about it than I do.

Same way that if you make 2 "normal" GETs simultaneously to the same server the responses don't get mixed up, i.e. they'll be separate HTTP Connections which are separate TCP connections:

> Each HTTP connection maps to one underlying transport connection.

-- https://httpwg.org/http-core/draft-ietf-httpbis-messaging-la...

TCP uses "port numbers" to identify different connections. 2 different connections from your PC to the same web server will use 2 different "source" ports. A port is just a number in the TCP header. https://en.wikipedia.org/wiki/Transmission_Control_Protocol#...

HTTP/1.1 added pipelining, so you can make several requests on the same TCP connection before receiving a response. But the (complete) responses must arrive in the same order of the requests so it doesn't work for SSE.

HTTP/2 added request & response multiplexing on the same TCP connection. But (according to the OP) there are some limitations that affect SSE.

Thanks! I spent 3-4 months on it (getting up at 5am most days to spend 1-2 hours on it before my day job) so it's nice to know it was of some use! :-)

I suspect it's because so many Emacs tutorials out there cover the same material as the built-in tutorial. And so many pre-packaged Emacs configuration files disable the menu-bar for no good reason other than that's what everyone else seems to be doing.

e.g. "Emacs Prelude", one such configuration or "distribution" of Emacs, used to disable the menu-bar by default as recently as a year ago: https://github.com/bbatsov/prelude/commit/878cffb

Years ago I wrote the Emacs tutorial I wish had existed when I was learning Emacs: https://david.rothlis.net/emacs/tutorial.html

It follows a similar philosophy as the OP: Start by reading Emacs's built-in tutorial, and then my tutorial shows you how to leverage the built-in help system to figure out how to achieve what you want to do.

Hold on, 2 million divided by 1 million equals 2, not 0.000000014. What am I missing?

Ninja's "-d explain" is next to useless in a large build: It says what's dirty but it's hard to relate which dirty input caused which output to be re-built. At the beginning of your build you get pages & pages of "these things are dirty" and then it starts the actual build. Also, this "explain" output is printed before ninja has realised that a lot of those things aren't actually dirty (thanks to "restat").

I've been using this in my build system to make the explain output slightly more useful: https://github.com/ninja-build/ninja/pull/2067

It's just a quick hack that took me half a day (most of which was remembering how to C++ after 8 years...) but I think it makes "-d explain" much more useful.

Really, the "explain" implementation could do with an overhaul. I'd love ninja to serialise the build graph to disk (or maybe just the subset of the graph that actually needed rebuilding). This could be augmented with the reason why each rule was run, and possibly even with the command output for each rule. Then the existing ninja tools like "graph", "browse", and "query" could optionally use this graph instead of the "build.ninja" file, so that you can inspect what happened the last time you ran "ninja".

Realistically I'll never get around to implementing this, and my pull request above gets me close enough.

Another way is to set $GIT_SSH_COMMAND when cloning:

  GIT_SSH_COMMAND="ssh -i ~/.ssh/id_ed25519_company_a -o IdentitiesOnly=yes" git clone ...

and then set it in your checkout's .git/config for subsequent fetches & pushes:

  git config core.sshCommand "ssh -i ~/.ssh/id_ed25519_company_a -o IdentitiesOnly=yes"

> The GIL will still be optionally available as an interpreter startup-time option

Sounds like they’re going to make it opt-in (or opt-outable) if they do merge it.

This contamination of test data from the training data reminds me of "Overly Optimistic Prediction Results on Imbalanced Data: a Case Study of Flaws and Benefits when Applying Over-sampling" [1] where almost 50% of the 24 peer-reviewed studies that use machine learning based on a particular publicly-available dataset, were claiming near-perfect accuracy at predicting the risk of pre-term birth for a patient, but were actually testing (accidentally) on training data.

[1]: https://arxiv.org/abs/2001.06296