Hacker News Reader

It's indeed written in Go now, which indeed has GC, and ultimately all memory is freed ... but there are indeed some issues around intermediate retention of memory, taking more memory than one would have expected.

Some gains were made on https://github.com/johnkerl/miller/pull/1133 and https://github.com/johnkerl/miller/pull/1132

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Also re 'hopefully the distros will keep up' -- some do some don't. I hope the C-to-Go issue isn't too much of a speedbump but we should know a couple distros in.

The other day I ran across https://repology.org/project/miller/versions -- something autogenerated on the web; I don't maintain it. Anyway you can see most distros are on 5.10 but there are some farther behind ... I will probably start with Conda and Brew since these are more personal/interactive, then maybe Fedora and Ubuntu -- ?

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Indeed -- in Go technically it's "concurrency" not "parallelism" (something the Go folks are careful to articulate) but with multiple processors available, in effect you get multicore processing so calling it "parallelism" isn't far off :).

More specifically, Miller uses separate "goroutines" (more or less threads) -- 2 for input (one for raw byte stream ingest and one for forming records), 1 for each verb (sort then head would be two verbs), and 1 for output (records back to strings), all pipelined up. Some of the recent perf improvements came from splitting the record-reader into two concurrent goroutines like that. Some of the older perf improvements are from pipelining the verbs.

But yes, if you have say 16 CPUs and you launch 10, or 20, or 30 Miller executables -- in the C impl each would soak a single CPU and anything beyond 16, the OS would have to multitask things. With Miller 6, basically the same kind of thing except each executable will be trying to use more than one CPU if it can. If it can't, the Go runtime and the OS will multitask. And both do tend to handle this stuff gracefully and without tuning on the part of the user.

Also I'd point out that even with big files & deeper multi-verb processing chains, htop rarely shows over 250% CPU, maybe 350% for deeper chains -- the input and output processing typically take most of the time, and the verbs in between not as much.

When we run out of memory, systems crash; when we run out of CPU, things just take a bit longer. I.e. the oversubscription is a real but non-fatal issue, for C or for Go ...

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Agreed; I've never really liked C compiles, and the Go build process is dreamy.

Another win (besides more features, and recent performance improvements) -- the Windows version is now a snap. Now `mlr.exe` takes it rightful place alongside the Linux and Mac executables (https://github.com/johnkerl/miller/releases/tag/v6.0.0-beta). Go+Windows is heavenly & I'll never need to tweak MSYS2/Appveyor/DLLs/etc again. :) See also https://miller.readthedocs.io/en/latest/miller-on-windows/

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Agreed, very much a user tool -- once it's released most people will get it via brew/yum/apt/conda/chocolatey and compile time will not be relevant for most folks.

Miller has always been a user tool and will remain so.

I think the confusion arose because I did the port per se and feature-adds first (which has taken most of the time), and left analysis/optimization (even low-hanging fruit like output-buffering) until the very end -- https://github.com/johnkerl/miller/pull/786 for example merged just a couple days ago.

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Thanks -- my apologies for the sequencing of the dev work. I focused first on the port per se; second on feature-adds; leaving benchmarking, performance analysis, and optimizations (even some real low-hanging fruit) until last. The 10x number was indeed the case pre-optimization ... on https://github.com/johnkerl/miller/pull/786 you can see tabulated results where Miller 6 is now on par with Miller 5, and in some cases is far faster.

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

The 10x number was before improvements on https://github.com/johnkerl/miller/pull/786 et al. The earlier negative perf results were my fault, not Go's -- I was focusing initially on the port and feature development, leaving benchmarking and optimization until the end. That said, Go is a bit slower than C line for line; however, Miller 5 (in C) was single-threaded and Miller 6 (in Go) actively uses multicore. This is why complex processing chains now run much quicker in Go than in C -- due to multicore and pipelining which are much easier to do in Go.

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Indeed -- that was before final optimization work on https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains. See that PR for tabulated processing-time results.

john_kerlon Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON

Distros are still at 5.10 pending a few final issues to be resolved -- then 6.0 can be generally released. Hopefully in a month or so.

The most important release blocker (now resolved) is https://github.com/johnkerl/miller/pull/786 et al., thanks to which Miller 6 performance is now on par with Miller 5 for simple processing, and far better than Miller 5 for complex processing chains.

john_kerlon Ask HN: What kind of music do you listen to focus?

I like upbeat (not droopy) and not too many lyrics (which can distract the language-processing parts of the brain that I need to work). Trip-hop fits the bill well for me.

* Groove Armada * Parol Stelar * Wax Tailor * Emancipator * Thievery Corporation * Bombay Dub Orchestra

john_kerlon Coming-out party for Universal Databases

This would be exciting even if I weren't joining TileDB in a couple weeks -- I'm looking forward to see this from the inside out as well as from the outside in.

john_kerlon Miller 5.3.0 – data comments, documentation improvements, and bug fixes

Linkfix https://github.com/johnkerl/miller/releases/tag/v5.3.0

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

regarding binaries -- at https://github.com/johnkerl/miller/releases/tag/v5.0.0 there are now macosx, linux i686 static, and linux x86_64 static binaries

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

i very much like the background artistically, and it's unobtrusive for me, but it seems that on different monitors / browsers it's less so. in any case, the background is now white. thanks for the feedback!

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

docs updated at http://johnkerl.org/miller-releases/miller-5.0.0/doc/build.h...

and, https://github.com/johnkerl/miller/issues/131

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

More here http://johnkerl.org/miller-releases/miller-5.0.0/doc/why.htm... but TL;DR is Miller handles several different file formats; jq is for JSON.

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

x86_64 binary fixed -- thank you!

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

http://johnkerl.org/miller-releases/miller-5.0.0/doc/etymolo..., if that helps ;)

john_kerlon Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions

Sorry, that is intentional but could be more clear:

http://johnkerl.org/miller/doc/build.html#From_git_clone_usi...