Ruby seems fine for brew. Does this do anything else better? Ruby makes it easy to write recipes for it which is a huge boon for a package manager.
The main reason I find brew to be a bit slow is just a lack of parallelism with regards to downloads and installs. Rather brew does alternating and sequential D, I, D, I, D, I, when I wish it just kept downloading in the background when it is doing the installs. It would cut down the brew upgrade time by 30% or more at the cost of more disk space used during the process.
But this one qualm I have isn't a result of its language implementation at all.
(This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)
I have heard that before.
Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-)
(I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.)
(Among other technical challenges, like updating the P2P broadcast for each new bottle.)
Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done.
> Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
You should only re-distribute the original file that was downloaded and thus one can just advertise the original torrent that was downloaded.
But as you said earlier, brew is a point in time command and this BitTorrent solution would only really work if brew switched to an always-on service. And I am not sure that many people want to do that, although I am sure some would.
Yes, brew exits when it is done installing, nothing would need to change about that if you used BT protocol to speed up downloads. I'm sure you do have some helpful users who would volunteer to seed their cache though, which would become feasible.
> That's not the challenge. The challenge is distributing those updates.
The metadata goes in the formula alongside the current metadata (URLs and hashes.)
Kinda. You do create a new torrent, but you distribute it in a way that to a swarm member is functionally equivalent to updating an old one. Check out BEP-0039 and BEP-0046 which respectively cover the HTTP and DHT mechanisms for updating torrents:
https://www.bittorrent.org/beps/bep_0039.html
https://www.bittorrent.org/beps/bep_0046.html
If that updated torrent is a BEP-0052 (v2) torrent it will hash per-file, and so the updated v2 torrent will have identical hashes for files which aren't changed: https://www.bittorrent.org/beps/bep_0052.html
This combines with BEP-0038 so the updated torrent can refer to the infohash of the older torrent with which it shares files, so if you already have the old one you only have to download files that have changed: https://www.bittorrent.org/beps/bep_0038.html
(There’s also still the state/seeding problem and its collision with user expectations around brew getting faster, or at least not any slower.)
I could see institutional seeders doing it as a way to donate bandwidth though, like a CDN that's built into the distribution protocol instead of getting load-balanced to Microsoft's nearest PoP when hitting a GitHub `ghcr.io` URI like Homebrew does today. Or even better, use that as an HTTP Seed (BEP-0019) to combine benefits of both :)
(My skepticism around whether this makes sense for Homebrew might be obscuring it, but I’m generally quite a big fan of distributed/P2P protocols, and I strongly believe that existing CDN decencies in packaging ecosystems are a risk that needs mitigating.)
There is no state/seeding problem. The client downloads from the same https url as always but uses peers on an as-available basis to speed things up and reduce load on the origin.
> The client downloads from the same https url as always but uses peers on an as-available basis to speed things up and reduce load on the origin.
So some kind of hybrid scheme, which (to me) implies the worst of both worlds: clients are still going to hammer upstreams on package updates (since client traffic isn’t uniform), and every client pays a bunch of peering overhead that doesn’t pay off until the files are “hot.” In other words, upstreams still need to plan for the same amount of capacity, and clients have to do more work.
(The adjacent thread observes that none of this is necessary if CDNs or other large operators do this between themselves, rather than involving clients. That seems strictly preferable to me.)
Using BitTorrent is also a great way to get banned from company-owned laptops.
And the language doesn’t have much to do with that. This project looks to be someone just toying around with Rust or their own PM. Props for that, but the headline has extra implications on HN.
I recently rewrote a big portion of Atlas [1]. It’s a Nim based dependency manager that clones Nim packages to a `deps/` folder. Initially I was worried about using reference types, etc, for performance reasons. It’s a general habit of mine. Then I remembered that stuff would be negligible compared to the download times and git overhead. Well aside from the SAT solver.
What part of the home brew experience do you find slow?
eg...
% time brew upgrade
brew upgrade 0.75s user 0.16s system 68% cpu 1.337 total
% time brew list
brew list 0.01s user 0.02s system 57% cpu 0.054 total
If this is true, why are the Rust tools, on average, so much faster than the JS (or whatever) tools they replace?
Hint: the answer hasn't just to do with the theoretical technical merits of the underlying platform, but also what type of developer they attract.
To be fair I haven't noticed Brew being as tediously slow as Pip. Maybe I just use it way less.
1. Python packaging, unlike Homebrew, does have a compute-heavy phase in the form of dependency resolution. `uv` can make significant gains over `pip` in that phase.
2. `uv` performs parallel downloads and other operations, in part because of Rust's fearless parallelism.
Homebrew doesn't really have (1), since resolution is just a linearization of the dependency tree. And parallelism of downloads in (2) poses a problem for Homebrew that's been mentioned in other threads (whereas parallelism is not a significant problem for PyPI's CDN-fronted distribution).