cyphar

Turns out that json.Encoder always appends a newline but (as GP said) json.Decoder will not attempt to read it if the JSON is complete at the power-of-two byte boundary (IIRC the buffer doubles for the first few iterations). The extra byte in the pipe when closed then causes reads and writes to go boom. We switched to using json.Marshal because it doesn't output a newline (sadly the encoded form isn't streamed as a result).

I still get twitchy around encoding/json as a result of that experience.

[1]: https://github.com/docker/docker/issues/14203#issuecomment-1...

The intended audience is obvious from first paragraph alone:

> At Anthropic, coding is no longer the bottleneck to building systems or doing research. Claude is able to autonomously iterate on research loops, micro-optimize complex programs, and build new services from the ground up at a speed and scale we haven’t seen before. The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI.

Translation:

> We are so far at the cutting edge we need to build shiny new infrastructure to replace the smelly "human scale" cruft. How could you /not/ value us at $2T?

In Rust "tuple unpacking" is really just a special case of pattern matching, which is a far more powerful concept and was actually taken from OCaml.

There actually is another way, but it isn't really that handy (as it is basically just shorthand for multiple assignment):

    func mustInt(v int, err error) int {
        fmt.Printf("got: (%v, %v)\n", v, err)
        return v
    }
    
    func foobar() (int, error) {
        return -123, nil
    }
    
    func badbeef() (int, error) {
        return nil, fmt.Errorf("moo")
    }
    
    func main() {
        vals := []any{
            mustFn(foobar()),
            mustFn(badbeef()),
        }
        fmt.Printf("got vals: %v\n", vals)
    }

    func mustFn[T any](v T, err error) T {
        fmt.Printf("got: (%v, %v)\n", v, err)
        return v
    }
    
    func foobar() (int64, error) {
        return -123, nil
    }
    
    func deadbeef() (string, error) {
        return "weird name for a cafe", nil
    }
    
    func badbeef() (*int, error) {
        return nil, fmt.Errorf("moo")
    }
    
    func main() {
        vals := []any{
            mustFn(foobar()),
            mustFn(deadbeef()),
            mustFn(badbeef()),
        }
        fmt.Printf("got vals: %v\n", vals)
    }

When I last did astrophysics research half a decade ago (as an undergrad) the workflow for doing research boiled down to writing pipelines to clean the data and then have scripts to find interesting candidates and analyse them automatically. It was necessary to filter stuff to reduce false positives, so it's possible this candidate was just missed by such scripts.

It's also possible this star is in some PhD student's dataset and they just haven't published yet. Astrophysics papers in this vein usually focus on techniques for classification and data science so you'd expect to have at least dozens if not hundreds to thousands of interesting stellar objects before you have something worth publishing unless it was some truly novel technique. A paper just saying "we found an exoplanet" isn't particularly interesting these days, as it turns out there are a lot of them.

(I haven't looked at the data and I'm sure my skills have atrophied by now anyway, sadly.)

More power to the guy who did this (astrophysics is probably the most hobbyist-friendly science since a lot of the data is public and the only lab equipment you need is a laptop) but I suspect "ordinary" folks are far more impressed by this than astrophysics folks. A decade or two ago exoplanets were still thought to be very rare but we've since figured out they are quite common and have moved on to cataloguing them.

But there is a bigger point to be made: in my experience most privacy activists argue that taking an absolutist stance personally is counter-productive -- just because it is almost impossible to completely eliminate risks from your life does not mean you should give up and not try to minimise them. And of course, the goal for most people isn't to protect against targeted attacks (since that means you already need to take fugitive-on-the-run-level precautions), it's to avoid becoming a target in the first place.

In case you were not aware, spooks have focused on metadata analysis over "full take" (data) analysis for a very long time because they are far more effective and require less data to analyse. This has been the case at least since the early 2000s with Stellar Wind but arguably even the Stasi worked this way -- they cared a lot more about who you were talking to than what you were talking about. The Snowden revelations in 2013 talked almost entirely about metadata-only systems that were being used to invasively surveil the world.

If your point was more "I have nothing to hide" then you can find plenty of articles online to disabuse you of that notion. There's even a Wikipedia article about it[2].

[1]: https://abcnews.com/blogs/headlines/2014/05/ex-nsa-chief-we-... [2]: https://en.wikipedia.org/wiki/Nothing_to_hide_argument

Even better, the OCI distribution protocol (which was Docker distribution until a few years ago) is not a static-blob-over-HTTP protocol! The blob bits are and you can route them to blob storage but you need a smart server for a few key bits of the protocol.

Back in the day I made proposals for distribution formats that didn't have these flaws and were more distributed (and previous proposals like AppC's discovery had similar ideas) but they were roundly rejected by the Docker people.

Nix does have a similar model in some respects (in particular the idea of input-hashed build graphs) and so solves a lot of similar problems but there were a few important factors that made us not go with Nix (or NixOS) and instead go with Buck2.

As the blog post mentions, the ability to build directly from a wide variety of upstream distribution artefacts and maintain temporary deltas was incredibly important to us. Our team has a lot of old-school packaging experience (I've done packaging for (open)SUSE for over a decade now and we have former and current Fedora, RHEL, Ubuntu, and Debian packaging folks with even longer histories) and our view is that distributions do a lot of thankless and incredibly valuable work and the most painful part of maintaining a distribution is the deltas you have to maintain forever after you've hard-forked packages or repackaged them. Buck2 allows us to create models for these concepts (which are in tine) without needing to boil the ocean repackaging upstream packages. But Daan and Martin are far better equipped to get into the weeds on this topic.

On the more general topic of why not use NixOS, as discussed in one of our other recent blog posts[1] we are make heavy use of sysexts and the /nix/store combined with the general linking model of NixOS (while very technically impressive) does not play particularly well with that model. I should also mention that we have a NixOS guy on our team too so you'd better believe NixOS came up in conversation. :D

[1]: https://amutable.com/blog/it-starts-upstream-kernel#image-ba...

The same can be said of C or C++ constructs (and many "anti-Rust" people have historically said that) -- the point is that their use is not enforced by the language and so bugs can lead to panics.

I write a fair amount of Go and Rust so I really don't think either language's flaws are fatal, but it comes off as weirdly defensive to redefine memory and data safety to be "well if you use it properly it's safe". It's totally fine to say this is a problem the Go language did not find important enough to require compile time enforcement and so solving it is done by convention and testing with the race detector (which a similar answer C and C++ give to this problem).

Sadly, almost all language standard libraries do not provide the right abstractions for dealing with files (the primary focus is on global paths as opposed to scoped paths or file descriptors / file handles) so it's little surprise bugs like these just keep popping up every few weeks.

To eliminate these from your codebase you need to rethink and really focus on being aware of how you deal with files. If the program you're writing has root privileges then you need to be more careful about misdirected writes to /proc or other pseudofilesystems.

[1]: https://github.com/cyphar/libpathrs

Truly the worst of all worlds.

A full rundown would be too long but [1] is a pretty good video going through the history of flight and how most of the other candidates for "first flyers" people bring up don't count and how it's difficult to argue that any of their contemporaries were even close to the Wrights.

[1]: https://www.youtube.com/watch?v=EkpQAGQiv4Q

Yes, datacentres can be reused for other purposes but dark fibre didn't save the telecom bubble and the utility of train tracks didn't save the railway bubble.

* of course it's not unlimited unlimited but I've not heard of anyone being cut off.

As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.

MapleStory was also quite popular back then for similar reasons.

In Australia we pronounce the H more like "heitch" while in the US people tend to pronounce it as "eytch" (IPA would probably more descriptive here, sorry!). In the UK it's a bit more of a mixed bag, as others have already elucidated.

"Ambiguous" was probably a poor word choice, I meant to say that because both ways of writing are technically valid based on the accent of the author there is an element of freedom in the choice that tells you something about them.

Those are also interesting but it feels a bit more special to me when it's a secondary speech characteristic because it feels a bit more like a clue in a detective novel -- it almost feels like you can hear inside the author's head and how they vocalise while they write.

In this case, "missle" is a malapropism that is only possible if you don't pronounce the second "i", and so is a secondary "speech" characteristic.

It is always fun when you notice where someone is from based on secondary clues like that -- my favourite is the choice of "a" or "an" where it is ambiguous (for instance, I recently ran across "an HTTP request" where I would've said "a HTTP request").

Well, PBS didn't do the best they could. Aside from the whole "one backup is no backup" problem, they should've immediately made a new backup as soon as Open Source Storage didn't want to renew their contract and stopped responding to them.

That's the most screaming-red, horn-blaring, strobe-flashing warning that they were about to go under and it was the job of the PBS CTO (who makes $180k/yr) to recognise that and respond immediately (i.e., send an intern the nearest Best Buy to buy 70TB of HDDs, they'd probably still have enough space in their bag to do a coffee run on the way back).

> Wasn’t there an avenue for the CEO to send a letter to Iron Mountain mapping the data and its owners or make an arrangement to legally transfer the data or something?

And if they made a mistake in providing the mapping (given they were a company going bankrupt, they might have trouble getting staff to do a complete audit of their customer data)? I would be surprised if the CEO would not be personally on the hook if they violated HIPPA or some other privacy requirement by allowing PBS access to something by accident.

Once a company is in dire straits like this, sussing out problems like this is one of the main purposes of the legal system. Ironically, 70TB of HDDs probably cost less than the first demand letter PBS sent to Iron Mountain.

If so, it would be incredibly reckless for Iron Mountain to provide all of the data stored by Open Source Storage to one of Open Source Storage's former customers (PBS). Presumably that is why they wanted to go through the legal process so that someone else would be vested with the legal authority to determine what data PBS can legally access.

Someone committing crimes "on behalf" of a company is not protected by the corporate veil and absolutely will get prosecuted individually (well, for small companies -- big companies get to play with different rules, which was the point GP was making).

Also the corporate veil is not absolute, it can get pierced in a few cases, and not everyone incorporates when they do contract work (in which case you need indemnity / private liability insurance).

In fact there is a simple way to do it -- add contradictory axioms and then you can use the principle of explosion to prove any statement as true. Is such a system inconsistent and thus useless? Yes, but it is complete.

But traits wouldn't really make sense for Go -- my point was that there is are solutions for this problem in other languages; Go's original design goals don't gel well with generics which leads to these kinds of issues. They felt generics weren't necessary so they didn't design Go with them in mind.

I was a little surprised how often I ran into this issue when using libraries that started defining structures as generic containers.