rwiggins

It's certainly bad form to turn a GitHub reply into a blog post(?). (Disclosure: I only saw the commit refs that say "blog post", that appear to be a reply.)

As a maintainer (but not on this project), I feel an implicit (professional) responsibility to carefully review PRs, especially from random people. And as a profession, we know it's often harder to read code than it is to write it. This goes double for LLM code, which sometimes lacks a coherent mental model.

So in general, the asymmetry of "I did very little work here, but I'd like you to spend your time carefully reviewing it nonetheless" can be very annoying.

But to talk about specifics: this PR's description is mostly a waste of time. The first section is a pure duplication of the diff. If I wanted before/after code blocks, I'd click the "Files Changed" tab. Ditto for "Docs-only: 4 insertions, 1 deletion, docs/graph.md only. No change to the player bundle, layout or any other page." - yeah, I can see that in the diff, thanks. (Well, that's what it was in the original diff, before the follow-up changes.)

The testing and AI disclosure sections are similarly overly verbose. I don't need commentary on the policy, nor do I need a statement permitting me to reject the PR.

As an outsider, yeah, my reaction is pretty negative. I'd much prefer a description like "Fixes #3224. Migrated from v2 asciinema-player to v3 AsciinemaPlayer.create, as described in v2 to v3 migration guide [link]. Tested locally as best I could."

A maintainer needing to ask to migrate the other v2 uses isn't a great look, either. The follow-up "fixed the other 5" comment could simply end at "fixed the other 5", without all of the nonsense afterward.

I have to wonder whether all of this truly saved anyone time.

Have you tried running the same review skill back to back in fresh sessions? My experience has been: 1) build complicated thing with LLM, 2) run code review, 3) it finds like 8 things, 4) fix them, 5) run another code review, 6) it finds another new 8 things, rinse and repeat.

I'd guess people truly are finding things pre-review. It's just that LLM review seems to have a limit to the number of problems it can or will find. While at the same time, LLM-written code can be an almost limitless source of bugs and errors.

It has no sense of when an architecture is doomed and needs fundamental changes. Instead, it will happily continue to chase individual bugs nearly to infinity.

But, maybe your review skill is better than the ones I've tried.

Hmm. I stick to jq for basically any JSON -> JSON transformation or summarization (field extraction, renaming, etc.). Perhaps I should switch to scripts more. uv is... such a game changer for Python, I don't think I've internalized it yet!

But as an example of about where I'd stop using jq/shell scripting and switch to an actual program... we have a service that has task queues. The number of queues for an endpoint is variable, but enumerable via `GET /queues` (I'm simplifying here of course), which returns e.g. `[0, 1, 2]`. There was a bug where certain tasks would get stuck in a non-terminal state, blocking one of those queues. So, I wanted a simple little snippet to find, for each queue, (1) which task is currently executing and (2) how many tasks are enqueued. It ended up vaguely looking like:

    for q in $(curl -s "$endpoint/queues" | jq -r '.[]'); do
        curl -s "$endpoint/queues/$q" \
        | jq --arg q "$q" '
            {
                "queue": $q,
                "executing": .currently_executing_tasks,
                "num_enqueued": (.enqueued_tasks | length)
            }'
    done | jq -s

which ends up producing output like (assuming queue 0 was blocked)

    [
        {
            "queue": 0,
            "executing": [],
            "num_enqueued": 100
        },
        ...
    ]

I think this is roughly where I'd start to consider "hmm, maybe a proper script would do this better". I bet the equivalent Python is much easier to read and probably not much longer.

Although, I think this example demonstrates how I typically use jq, which is like a little multitool. I don't usually write really complicated jq.

Oh wow, that's fantastic. I love that it includes real values while still summarizing the doc's structure. I'm going to steal that. I'll probably keep jq-structure around because it's so easy to copy/paste paths I'm looking for, but yours is definitely better for understanding what the JSON doc actually contains.

Oh, fantastic. jq has become an integral part of work for me.

I'll use this opportunity to plug the one-liner I use all the time, which summarizes the "structure" of a doc in a jq-able way: https://github.com/stedolan/jq/issues/243#issuecomment-48470... (I didn't write it, I'm just a happy user)

For example:

    $ curl -s 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]'
    .
    .createDate
    .ipv6_prefixes
    .ipv6_prefixes[]
    .ipv6_prefixes[].ipv6_prefix
    .ipv6_prefixes[].network_border_group
    .ipv6_prefixes[].region
    .ipv6_prefixes[].service
    .prefixes
    .prefixes[]
    .prefixes[].ip_prefix
    .prefixes[].network_border_group
    .prefixes[].region
    .prefixes[].service
    .syncToken

(except I have it aliased to "jq-structure" locally of course. also, if there's a new fancy way to do this, I'm all ears; I've been using this alias for like... almost a decade now :/)

In the spirit of trying out jqfmt, let's see how it formats that one-liner...

    ~  echo '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]' | ~/go/bin/jqfmt -ob -ar -op pipe
    [
        path(..) | 
        map(if type == "number" then "[]" else tostring end) | 
        join(".") | 
        split(".[]") | 
        join("[]")
    ] | 
        unique | 
        map("." + .) | 
        .[]%
    ~  

Not bad! Shame that jqfmt doesn't output a newline at the end, though. The errant `%` is zsh's partial line marker. Also, `-ob -ar -op pipe` seems like a pretty good set of defaults to me - I would prefer that over it (seemingly?) not doing anything with no flags. (At least for this sample snippet.)

> how clear and useful it is. Perhaps, but do you really need all that?

Do I need clear and useful things? Maybe not. Would I like to have them anyway? Yes.

Years ago, I configured a Java project's logging framework to automatically exclude all the "uninteresting" frames in stack traces. It was beautiful. Every stack trace showed just the path taken through our application. And we could see the stack of "caused-by" exceptions, and common frames (across exceptions) were automatically cut out, too.

Granted, I'm pretty sure logback's complexity is anathema to Go. But my goodness, it had some nice features...

And then you just throw the stack trace in IntelliJ's "analyze stacktrace" box and you get clickable links to each line in every relevant file... I can dream.

> the wrapping is very greppable when done well

Yeah, that's my other problem with it. _When done well._ Every time I write an `if err != nil {}` block, I need to decide whether to return the error as is (`return err`) or decorate it with further context (`return fmt.Errorf("stuff broke: %w", err)`). (Or use `%v` if I don't want to wrap. Yet another little nuance I find myself needing to explain to junior devs over and over. And don't get me started about putting that in the `fmt` package.)

So anyway, I've seen monstrosities of errors where there were 6+ "statement: statement: statement: statement: statement: final error" that felt like a dark comedy. I've also seen very high-level errors where I dearly wished for some intermediate context, but instead just had "failed to do a thing: EOF".

That all being said, stack traces are really expensive. So, you end up with some "fun" optimizations: https://stackoverflow.com/questions/58696093/when-does-jvm-s...

It's been so long that I barely remember the details, but yes, we used a gnomon to calculate several things... the earth's circumference, our local solar noon, latitude and longitude, etc.

I specifically remember that we measured how far the shadow moved over time using chalk. I think this was in lieu of having a second triangle somewhere else, although, for all I know we might have used a second reference distance.

Our campus had this elaborate outdoor "observatory" that had all sorts of interesting features. The gnomon was one of them, but there were other cool things, like IIRC there was a metal sculpture where Polaris would be seen through a central hole (and a guide inscribed in the concrete to figure out where to stand for that to happen -- I think it was based on viewing height).

> Have you ever done the experiments to prove the Earth is round?

I have, actually! Thanks, astronomy class!

I've even estimated the earth's diameter, and I was only like 30% off (iirc). Pretty good for the simplistic method and rough measurements we used.

Sometimes authorities are actually authoritative, though, particularly for technical, factual material. If I'm reading a published release date for a video game, directly from the publisher -- what is there to contest? Meanwhile, ask an LLM and you may have... mixed results, even if the date is within its knowledge cutoff.

Setting aside that you can enable mouse mode in vim and use the mouse to your heart's desire --

and setting aside the "declarative" vs "imperative" nomenclature debate --

vim is pretty dang efficient. Typing `gg` to go to the beginning of a file, or `G` to go the end. Or all the other variety of motions. There are a lot.

Also, there is not necessarily any need to move the cursor anywhere. For example, if I'm in the middle of writing some string and I've decided to change the whole thing, `<ESC>ci"` does that, with no cursor movement required.

Yeah, having the read the article, my conclusion was that you should avoid honeycrisps from like February through ~August. i.e. only buy them when they're vaguely in season, or not too long afterward.

Anecdotally, I had some pretty delicious honeycrisps last night (in WA).

Errr, not in the rural area I grew up in. Gravel driveways are super common, gravel roads not so much.

To give some specifics: I only remember driving down an actual gravel road (like, for public use) a single time. In 18 years. Even my friends who lived >30min from the nearest "city" (~10k population) had paved roads all the way.

But that is just my own experience. Areas with a different climate or geography might be a totally different story. My hometown area is relatively flat, lots of farmland, and rarely gets severe winter weather.

Maybe area-dependent? I grew up in an extraordinarily rural area in Tennessee. Most roads were paved (asphalt). Even ones out in the middle of nowhere.

The conditions of some of the remote roads might not have been great, mind you... and some seemed "thinner" almost, maybe paved a long time ago?

I felt the same way initially, but the language has grown on me. The turning point was writing a lot of Go, like full-time project work for a few months.

But as I've gotten older, I've started striving more and more for simplicity above all else, especially in systems design (disclaimer: I'm an SRE). Go is pretty good at being simple.

There are some things that still annoy me a whole bunch, though. Like - just one example - `fmt.Errorf` not being a first-class syntactic construct (or the difference between `%v` and `%w` in `fmt.Errorf`).

It's a link to source code on GitHub. It could be marketing too, but it's absolutely, incontrovertibly source code.

> show us the code if it's that easy to give to others

See https://security.apple.com/documentation/private-cloud-compu...

disclosure: work at Apple, opinions absolutely my own.

I haven't used Terraform in years (because I changed jobs, not because of the tech itself), but back in the day v0.12 solved most of my gripes. I have always wished they'd implement a better "if" syntax for blocks, because the language itself pseudo-supports it: https://github.com/hashicorp/terraform/issues/21512

But yeah, at $previous_job, Terraform enabled some really fantastic cross-SaaS integrations. Stuff like standing up a whole stack on AWS and creating a statuspage.io page and configuring Pingdom all at once. Perfect for customers who wanted their own instance of an application in an isolated fashion.

We also built an auto-approver for Terraform plans based on fingerprinting "known-good" (safe to execute) plans, but that's a story for a different day.

Out of curiosity, how would you describe TCP in these terms? Does the TCP stack's handling of sequence numbers constitute processing (on the client and server both I assume)? Which part(s) of a TCP connection could be described as delivery?

Page not found - is that a private repo?

The source is on GitHub:

utils.js, which contains the (de)serialization code: https://github.com/ccbikai/loooooooooooooooooooooooooooooooo...

tool.js, which does the serialization: https://github.com/ccbikai/loooooooooooooooooooooooooooooooo...

display.js, which does the deserialization and redirect: https://github.com/ccbikai/loooooooooooooooooooooooooooooooo...

Unlike SO, it's common to have very situation-specific questions posted on YAQS. In fact, my team preferred random one-off questions to go through YAQS (our contact golink pointed you to a monitored YAQS queue) precisely because they're much more searchable (and scalable) than point-to-point chats.

So yes, searching for your GShoe error, and (assuming you found nothing) asking about it on YAQS is not a bad way to get help from some random faraway team.

I suppose it's partially because most team chats are locked down (invite-only). In a company with a reasonably open slack, you might be able to ask in #gshoe-team or search it for relevant conversations, but not at Google in my experience - and this is setting aside the issue of message retention.

BTW, I agree 24h retention was truly ridiculous. Most of my colleagues hated it - fortunately (probably as a result of this legal case!) they disabled it and now the default is 30d everywhere.

Regarding promo, community contributions are still very much an expectation. Being active on YAQS counts toward that. True, the promo committee isn't going to go looking for it, so your manager needs to agree YAQS is a level-appropriate community contribution and include that in your promo packet.

Disclosure: I left Google like, a couple weeks ago

Exactly this. It's not uncommon to ask a coworker how to do something, get a response, and then send them a follow-up CL adding that information to a playbook or doc somewhere so others can reference it.

Heck, sometimes I responded to questions with a CL adding that info to docs.

The internal search engine helps drive a lot of this, too - if you want to know how to do something, docs (via search) are like the #1 choice. So, everyone's pretty incentivized to make it a good resource.

Best/worst. I worked at an e-learning company that, thanks to ancient e-learning standards, really tried hard to figure out when a user is leaving a web-page to handle session exits gracefully.

sendBeacon was dead on arrival, for that purpose at least; iirc it's blocked by most ad blockers. (Or was, who knows with Manifest v3.)

Similarly we're unlikely to see sendBeacon truly replace tracking redirects, unfortunately. This isn't an area where tech firms are happy to see fine user controls.

There's a fourth use-case: occasionally, gaming.

I play Final Fantasy XIV, an MMORPG - apparently, supposedly, the peering connection between AT&T and FFXIV's US ISP (NTT) was particularly bad. [1]

This manifested as pretty severe connection issues for AT&T customers playing FFXIV. Except, it was a chronic issue that would only flare up when that particular connection point was stressed.

One of the easiest workarounds? Hop on a VPN.

That's one example. Anecdotally, I have a few friends that toggle VPNs on and off when they encounter "network weather" in games. Personally, I'm a bit skeptical they're truly so often mitigating problems by toggling a VPN (instead of, say, just waiting a couple minutes), but hey, they swear by it.

[1]: https://forum.square-enix.com/ffxiv/threads/482155-Bad-lag-a...

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.

I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.

That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.

[1]: https://github.com/siderolabs/talos/blob/main/internal/app/m...

[2]: https://www.talos.dev/v1.6/reference/cli/#talosctl-list

It has indeed been a strange time for Google SRE recently. However, they're definitely not planning on shutting down SRE - at least, if you can trust what Google leadership's actual explanation of what that meant.

Supposedly, the ratio of SRE to product eng had been growing slowly over the years. The KR to "readjust" that ratio was to bring it back in line with historical norms, i.e., to ensure that SRE continued to scale sub-linearly with SWE/systems. This had (primarily) two facets.

First, it gave SRE teams an effectively-blank check to reevaluate their existing dev engagements and jettison the ones that weren't working well.

Second, it pushed to eliminate old tools/systems/platforms and converge onto the more modern stuff, like Annealing [1]. Fewer crufty platforms means fewer teams needed to run them, and improvements in those platforms have broad impact.

Anecdotally, my own sub-org (within SRE) is growing at the moment. Not by a huge amount, but growing nonetheless.

[1]: https://www.usenix.org/publications/loginonline/prodspec-and...

GCE local SSDs absolutely are on the same host as the VM. The docs [0] are pretty clear on this, I think:

> Local SSD disks are physically attached to the server that hosts your VM.

Disclosure: I work on GCE.

[0] https://cloud.google.com/compute/docs/disks/local-ssd

There's something super weird going on with those line numbers. They don't align for me either (Firefox on Windows 11). In the web inspector, the line numbers are rendering with Courier New as the font, whereas the code itself is `monospace`.

The weird thing is both are controlled by a single CSS rule,

    div#cgit pre {
      font-family: "Source Code Pro", "Courier New", monospace;
    }

(Changing to just `monospace` fixes it.)

I'm not sure how two elements could have different fonts with that. I'm possibly missing something obscure, but it really feels like a browser bug.

You know, in retrospect, I think Kagi expects O(thousands) searches per month per user, so doing per-user usage accounting in the DB is fine -- thanks to row-level locking.

Well, at least until you get a user who does 60k "in a short time period"... :-)

Aaaahhh, it's crazy how much this incident resonates with me!

I've personally handled this exact same kind of outage more times than I'd care to admit. And just like the fine folks at Kagi, I've fallen into the same rabbit hole (database connection pool health) and tried all the same mitigations - futilely throwing new instances at the problem, the belief that if I could just "reset" traffic it'd all be fixed, etc...

It doesn't help that the usual saturation metrics (CPU%, IOPS, ...) for databases typically don't move very much during outages like these. You see high query latency, sure, but you go looking and think: "well, it still has CPU and IOPS headroom..." without realizing, as always, lock contention lurks.

In my experience, 98% of the time, any weirdness with DB connection pools is a result of weirdness in the DB itself. Not sure what RDBMS Kagi's running, but I'd highly recommend graphing global I/O wait time (seconds per second) and global lock acquisition time (seconds per second) for the DB. And also query execution time (seconds per second) per (normalized) query. Add a CPU utilization chart and you've got a dashboard that will let you quickly identify most at-scale perf issues.

Separately: I'm a bit surprised that search queries trigger RDBMS writes. I would've figured the RDBMS would only be used for things like user settings, login management, etc. I wonder if Kagi's doing usage accounting (e.g. incrementing a counter) in the RDBMS. That'd be an absolute classic failure mode at scale.

I like 'BigTech' -- it just sidesteps this issue.