Hacker News Reader

Comment by woodruffw | Hacker News Reader

To copy my response on Lobsters:

(I am one of uv’s maintainers.)

The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.

With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?

(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)

View on HN

> The nice thing about Python packaging is that it’s standardized

But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.

Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.

> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.

As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.

> I don’t understand why Gentoo needs to rebuild every single release of uv-build.

Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.

> you can do a normal version bound in your Python build backend metadata

Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".

I linked to some of the relevant standards below.

At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.

(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)

> it seems extremely unlikely that they actually need to build every single patch version of uv.

If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.

distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.

Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again.

So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.

Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.

i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time.

my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.

for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...)

This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?

> i guess gentoo is rolling-release

That's my understanding, yes.

> my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.

Thing is, that apparently won't work. From [0], with some more discussion starting at [1]

> > I don’t understand why Gentoo needs to rebuild every single release of uv-build.

> Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.

[0] <https://news.ycombinator.com/item?id=50077466>

[1] <https://news.ycombinator.com/item?id=50077553>

Per my response, that isn’t how Python packaging works. There is no “use the latest version” mechanism in standard Python packaging; leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.

> ...leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.

So, would it be correct to decide that package with an unconstrained version gets built with 0.1.0? And when that fails to build, what's the distro packager to do? Bang on the package maintainer until they specify the real constraint? Yeah, good luck with that.

Doesn't it make way more sense to assume that "unconstrained uv-build version" means "they only tested this with the very latest version of uv-build" and proceed accordingly? This will fail sometimes, but I figure it'll fail far less often then assuming that you get to use 0.1.0 if no version is specified.

I'm being a little extreme with my example here, but only to illustrate the larger point I'm making. Breaking changes do happen, and without specified version constraints one has to assume that "very latest available" is the correct one.

Well, that’s why you should use a semver range, as is the default, instead of leaving it unconstrained. That’s the actual solution that solves the problems for all parties here; I’m just pointing out that “unconstrained = latest” isn’t a correct summary of how Python packaging works.

In that case, wouldn't it be better for uv to reject unconstrained? Gentoo (or any packager, such as homebrew / nix) is too far downstream for that. Presumably allowing unconstrained seemed to make sense in some circumstances at some point?

No, because it’s a standard part of Python packaging, and it’s not the problem here.

(There are all kinds of legitimate reasons to use an unconstrained dependency, including locking it separately and also trying to help other downstream Linux distros with their own bespoke packaging problems.)

you literally just said

  > that’s why you should use a semver range, as is the default, instead of leaving it unconstrained

You should probably pick one of these two mutually exclusive options and stick with it.

> Gentoo can't second guess you

gentoo appears to support exactly 7 versions of uv-build, a list they update by hand: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...

git changelog of uv-build in the gentoo package manager: https://gitweb.gentoo.org/repo/gentoo.git/log/dev-python/uv-...

gentoo also wrap python packages in their own package manager, packages in this tree appear to come from their upstream version control system eg Github, not PyPI.

just because someone on hn said something, doesn't make it true.

> they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection

I don't follow. This feels like Gentoo is mishandling build-tool dependencies.

Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?

Strictly speaking, they should, right? Differences in GCC could produce differences in anything it compiles.

IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed.

No, and not with clang or llvm or rust. When we get a new Golang tho, that does rebuild stuff built with Go. And big perl updates cause a rebuild for perl packages.

> I don't follow.

I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.

Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.

> This feels like Gentoo is mishandling build-tool dependencies.

I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.

> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?

No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.

> The build backend is an explicit part of the specification of a Python project

Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update - that's a behaviour they've decided is correct on their own.

> C/C++ libraries don't have an explicit specification of the build tool to be used in their project

They don't have a standard way to specify such, but they have an implicit dependency on the installed compiler toolchain, and it would be just as valid to interpret "no way to specify a build-system dependency" as "always rebuild" when the build tool updates

> Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update

Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update but nothing else in the project does. So what's a "build everything from source" distro to do? If it's possible that updating the build dependency will change the build artifact, doesn't it have to rebuild?

> Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update

With this framing, exactly the same thing can be said about libstdc++, for example, so must we also force a full system recompile on every GCC/clang version bump?

Seems like a design flaw in gentoo that previously hadn't been an issue because build tools weren't being updated nearly as often as uv-build is.

> Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.

Sure it does. TFA claims

> all Gentoo users need to build it from source, and given that it’s released quite frequently (12 releases in the 30 days prior to writing this post), that’s a lot of building. Like, for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.

And that is false, exactly because of that standard. If you want reproducible builds then you can stick with a previously built version of uv-build for a long time. If you want working builds on the fly using your own processes, then the standard empowers you to substitute something else. If you want to use the Python packaging ecosystem in the normal way, then it doesn't even matter what your distro is.

uv-build is for pure Python projects, which entails that the "build" process is trivial. It doesn't even bytecode-compile the .pyc files generally; wheels aren't expected to contain them (your installer may produce them at install time, or you can let Python produce them on demand). Binary reproducibility for pure-Python wheels is just not, from anything I've seen, a high priority for Python packaging.

Linux distros have these ideas about packaging that are just not suitable for Python and not in tune with what the Python ecosystem is doing. The vast majority of the Python code you install should not be integrated into your "system". (Python devs rarely find themselves wanting to "bootstrap" everything from source, either.) Gentoo's issue here appears to be their own policy, which is a problem of their own making. It's unrealistic for them to expect random Python devs to care about this.

> As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.

If you mean because of Gentoo policy, that's their fault. I can guarantee you that approximately nobody using uv-build is concerned with whether their package makes it into Portage. Python build systems aren't designed to produce packages native to Portage (or any other Linux packaging system). Probably the overwhelming majority of these users aren't expecting any Linux distro repositories to host their package in any form, and probably most of them couldn't even tell you what Gentoo is.

If you mean technically, then no; once you have the source code and are using ebuild, nothing compels you to choose the backend specified in pyproject.toml. If everything is up to standard then it's trivial to switch it over to, say, Flit. Most projects following a standard src layout will need no tool-specific config, and there's quite little of it possible regardless (and all well documented). The author even highlights:

> It’s a backend and the whole point of PEP 517 is that you can use any backend with any tool. You can use flit-core, you can use hatchling, and either of them will work just fine with uv (or any other frontend).

The edits do take time but it's a lot better than the proposed bogeyman of waiting for a new build of uv-build from Rust source.

> nothing compels you to choose the backend specified in pyproject.toml

This comes as a complete surprise to me. I thought the whole point of specifying the build backend in pyproject.toml was so the project owner can choose which build backend their project gets built with, and have that choice automatically enforced by tools.

Are you saying this kind of thing is allowed by the Python spec? If so, where? Can you give a specific document?

> The author even highlights

What he means is that the project owner doesn't have to, for example, use uv-build as their build backend in pyproject.toml just because they are using uv as their overall project management tool. I do not think he means that he, as a distro packager, is free to just ignore what's in pyproject.toml.

> the project owner can choose which build backend their project gets built with

When following an automated workflow implemented by an installer or other tooling, yes.

But you own the sdist file, and the archive contents, after downloading it. Nothing prevents you from editing pyproject.toml, using a build frontend/system that offers a flag to disregard PEP 517, running your own custom scripts on the data, manually copying the files into the target environment, or whatever else you can think of.

And since Gentoo is doing its own packaging with its own package format, nothing prevents them from doing the same kind of patching.

> you own the sdist file, and the archive contents, after downloading it.

If you're forking the project, sure. But I don't think that's what Gentoo is doing.

If you're trying to build what the project owner intended for you to build, I don't see how you can just ignore what the project owner specified. Nor do I see how that is allowed by the Python specs. That's why I asked you for a reference if you think it is in fact allowed by the Python specs.

If your position is that the Python specs don't allow you to just ignore pyproject.toml, but a distro can just ignore what's in the Python specs, I don't agree with the latter either, but it's a different position, and I'm not sure which one you're taking.

> uv-build is for pure Python projects, which entails that the "build" process is trivial.

The build process is more than just the Python code. There is also project metadata, which might be dynamically generated by the build backend. Indeed that's one of the main features offered by the different build backends, simplifying the project owner's life by removing the need to write a lot of boilerplate metadata that the tool can automatically discover on its own.

Yes, but these things are largely trivial in most cases.

In a wheel, there are basically two kinds of metadata: the kind that follow directly from the circumstances of the build (RECORD, which lists sizes and hashes for included files, and WHEEL, which says what environments the wheel is for and what tool built it), and then the kind that has to be specified by the developer. The latter is in METADATA. But the thing is, for most projects this data can be completely static, in which case the wheel METADATA is just a copy of the corresponding sdist PKG-INFO.

Build backends support a variety of dynamic metadata, but there are two places where that could be calculated: in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO) and when building a wheel from that sdist. The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.

> these things are largely trivial in most cases

The point is that that's not your call to make. It's the project owner's call. That's what the build backend specification in pyproject.toml is for. If you're going to just ignore that, what's the point of having it at all?

> in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO)

Which is the only place that's relevant for this discussion, since we're talking about Gentoo, which builds everything from source.

> The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.

I agree with all this, but as above, it's irrelevant to this discussion.

The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch.

Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...

I think that’s actually a different problem: packages that use uv-build for their backend should use a semver range, which enables the kind of “pick a version and use it” strategy I mentioned before.

(This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.)

> packages that use uv-build for their backend should use a semver range

This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change.

In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built.

That would appear to be the polar opposite of effective: that turns a global constraint problem (everybody needing to have or install a compatible build backend version) into a global coordination problem (everybody needing to have or install an exact version). I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.

It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.

> a global coordination problem (everybody needing to have or install an exact version).

This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place.

> getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.

You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway?

gentoo does not appear to download uv from PyPi https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...

It looks like it downloads from uv's git repository, which would also solve the coordination problem under discussion, since once a version is posted there it should remain there.

> I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.

This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them (or more precisely their users, since each Gentoo user is having to do this) if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates".

> if build backend versions change so often and projects don't pin a specific version but just say "use the latest"

This isn’t what’s happening: uv defaults to semver bounds, so a maintainer would have to go out of their way to introduce a troublesome backend constraint.

And note: there is no “use the latest” constraint in Python packaging. If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.

(I also know from first-hand experience that Gentoo and other distributions do not build Python packages as-is; they patch them in all kinds of ways. But that’s neither here nor there.)

> uv defaults to semver bounds

But a new patch release will be within the same semver bound, so it amounts to the same thing.

> If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.

What do build frontends actually do, though?

> But a new patch release will be within the same semver bound, so it amounts to the same thing.

Sorry, can you lay out the actual problem you’re envisioning for me? If uv-build goes from 0.13.1 to 0.13.2, then anybody with a semver bound of `uv-build>=0.12,<0.14` doesn’t need to update anything. They can keep using whatever version of the build backend they already have that satisfies that bound.

The main thing that would be troublesome is if a user pins their build backend to a specific range or version, e.g. `uv-build==0.13.1`. Then, upon bumping it, everyone who has already built 0.13.1 will need to invalidate and rebuild their build backend. But that’s why that isn’t the default.

> What do build frontends actually do, though?

uv will definitely use a compatible version. It won’t even download a copy of that that compatible version by default, since the uv binary itself is typically compatible and can invoke the build backend directly.

I’m not sure what pip and others do, but I would expect they do the same. It’s the obvious way to implement backend resolution.

> It’s the obvious way to implement backend resolution.

If a compatible version is already present, yes, a frontend can just use it if it's within the semver range specified.

But in many cases, builds are done in a fresh virtual environment that has nothing installed. In such a case, the only option is to download the latest version that's in the semver range specified. That's how the "build" frontend works, for example, unless you specify a flag to keep it from creating a fresh virtual environment.

Is that what Gentoo is doing here? I thought the whole point of the distro doing any sort of Python packaging is that they can pierce abstractions like virtual environments where it suits them.

(But also: if we’re talking virtual environments then we’re firmly in Python land, not distro land, where the correct answer is “use a wheel” instead of building it yourself.)

The whole point of Gentoo is to build everything directly from source. That means doing whatever the build source tells you to do. It also means not using a pre-built wheel, since that's not source code.

From what I can gather, Gentoo's Python build system does use a virtualenv [1]. Maybe that's not your definition of "distro land", but it appears to be Gentoo's.

It's not clear to me exactly what frontend build tool is used, whether it's one of the well known ones or something that the Gentoo developers coded themselves. However, if it's using a virtualenv, it's going to have to re-fetch the build backend and rebuild it.

[1] https://projects.gentoo.org/python/guide/basic.html

If it’s using a virtual environment, then they can pierce the abstraction by inheriting system packages, including a compatible uv-build. There’s nothing about Python packaging that requires them to rebuild the backend multiple times.

> If it’s using a virtual environment, then they can pierce the abstraction by inheriting system packages

Which defeats the purpose of doing the build in a virtualenv, which is to avoid any possible collision with or contamination by other stuff that's installed on the system by using a fresh copy of everything.

They’re already defeating the point of Python packaging in multifarious ways here. The entire reason uv and uv-build exist is because we have mature, smooth mechanisms for not rebuilding the world.

uv and uv-build are not equivalent to "Python packaging". Perhaps you would like them to be, but they're not.

I don’t know what “equivalent to” would mean here.

Speaking as someone who has written many Python packaging PEPs and who has maintained (and continues to maintain) PyPA tooling that you almost certainly use: uv and uv-build are as much a part of Python packaging as anything else that interfaces with the standards.

I hear you, but realistically the rustification of Python package internals is only going to strongly increase over the next year (I'm working on a number of such packages, myself), so systematic solutions to such annoyances are going to become increasingly necessary

Well, it did force me to work on better build caching and the like, but at the same time, it accelerated my wish to remove python where possible

This response downplays the portability concern which is the main point of the article. The ask for Gentoo to change its release & update cadence doesn't answer his plea.

As I see it, uv-build is Rust and will stay Rust, so SPARC or address-space-constrained 32-bit targets may never work right. Astral prioritizes its users who run on tier-1 targets, and the cost of their uv-init default lands on the downstream packages that Gentoo has to build. Gentoo wants projects to choose pure python backends to reduce that ecosystem load, Astral would prefer if more projects default to uv-build (IIRC the default went from hatchling to uv-build in mid-2025).

Long-tail & evolution of ecosystem would be a far more interesting discussion, but the broader community has a "we are the future/adapt to us" sentiment that forecloses trade-offs.

I mean, SPARC and anything else can work right. But what uv can’t do is conjure up support for platforms that the larger ecosystem won’t even claim to support.

To that extent, I think uv (and CPython, and Rust, and so many other projects) are essentially trying to be honest in saying “we have no way to build and test that to our own standards.” I think that kind of honesty in validation is its own kind of load reduction on the ecosystem, because it means less tacit assumption that the maintainers will support anything that happens to build.

The basic question the plea raises is "how should the long tail of the ecosystem get cut?".

"Upstream has no way to build and test [the long tail] to our own standards" is a certain kind of honesty but not a real answer or a trade-off based in the status quo ante. The status quo, as I glean it, is that C/autotools ecosystem let downstream packagers get from “it compiles” to “it runs” on their own; the new Rust-based one inhibits their freedom to do so.

That said, I respect Astral's decision to prioritize their product by changing their default regardless of the wider ecosystem; I just don't see the point of moralizing it.

Why? You can bootstrap Rust if you want, just like you can bootstrap LLVM or GCC. There’s no law or rule against it, there are just support tiers (and GNU projects have those support tiers too.)

(I don’t think it’s moralizing to do or say this.)

> The nice thing about Python packaging is that it’s standardized

And a problem with Python packaging is that it's far too flexible.

For a pure Python package, IMO every single file in the wheel should be bit-for-bit identical to a file in the source tree (with the possible exception of a manifest listing the files). If this were actually a guarantee, even just for a category of package that opted in, then:

a) The build "tool" could be replaced with a build file selector, which would often be completely trivial and could have default behavior that worked for almost every pure python package right in pyproject.toml and

b) Instead of the giant mess of sigstore and attestation and weird trusted builds we have today, we would have a straightforward proof that a wheel actually matched a git commit irrespective of who built it. In fact, the proof could be embedded in the wheel! And the proof could be verified without executing any package-controlled code.

Python is exactly as flexible as the people who use it want it to be. I suspect that most trivial solutions do not generalize to the interests of the broader Python community, which is how we end up with standards that are universally agreed to be not exactly what anybody wants.

Maybe, but I do think “pure Python sdists could just be a file selector” misses lots of common cases of simple code generation. Documentation boilerplate generators, version-at-runtime generators, things that generate e.g. static HTML or protobuf or whatnot results…it’s not most pure-Python packages, sure, but it’s a lot of them.

Fair enough.

As far as I know, packages that use or could use uv-build or flit-core are in the just-copy-files category.

Is Gentoo one of those compile-everything-from-source distros?

I'm pretty sure that is the entire purpose of Gentoo

Specifically, the purpose is a level of system configuration that is only possible by rebuilding stuff.

Yeah, it has binary packages nowadays as well, though.

They basically started it all, that was what made Gentoo different from everyone else.

It also made it nice base for both ChromeOS and its descendants like COS and Flatpak (continuation of original CoreOS)

I wasn't there when it started but wasn't the raison d'etre was the GCC fork that eventually became the new upstream for GCC. Because the fork was faster Gentoo got the reputation that compiling from source was to get faster executable. This is a misunderstanding imho, as the greatest strength of Gentoo the level of customization enabled by use flags.

I would not call egcs that fast, nor do I remember it having any role with Gentoo.

Gentoo was the first distro to use egcs as part of their default toolchain. Not sure if it was the only, but it was the only one I heard to had done so.

I can tell that in 2004, it was a regular GCC, and I got fed up compiling it.

Yes, and unlike NixOS they dont calculate build hashes for the sake of making binary caches. So truly, you have to build from source, even if your recipe is exactly the same as everyone else's. I've never really understood this zealotry around compiling things myself, and boy are there some big (slow to compile) programs out there these days.

There are binary packages in gentoo, but you can't take advantage of USE flags.

For one thing you can, in theory, optimise for the CPU you have instead of some generic target that has to run acceptably everywhere.

Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself. I know nixpkgs, for example, builds with very generic targets and not super aggressive compiler flags, but at least the machinery exists to do this.

On my 10 computers for example? :-) Gentoo wasn't aimed at people with such luxuries.

I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it.

> Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself.

Here are three links that you will find informative. [0][1][2]

While I'm not using the official Gentoo binhost, I've had one running on my LAN for my own personal use for roughly a decade now. It has been possible to set up a binhost for quite a bit longer than that... that's just when I got multiple machines with the same CPU in them, so it made sense for me to set one up.

[0] <https://wiki.gentoo.org/wiki/Binary_package_guide>

[1] <https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Bas...>

[2] <https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart>

And even before binhosts you could have a shared buildhost with ccache.

Was ccache support much older than binhosts? It has been so very long since the introduction of both that my memory of the history has been thoroughly erased.

Not sure, perhaps I was only aware of ccache before binhosts.

With apologies to Slackware, it is the compile-everything-from-source distro.

Happy user for over 23 years now!

Well, I got introduced to Linux via Slackware 2.0, and it already had binary tarballs.

I think Gentoo had a few (e.g. Open Office which took forever to compile). I don't recall if they did in 2003, when I started using it. But if not then, they had it soon after.

> The nice thing about Python packaging is that it’s standardized

Has anybody done the analysis to determine if this is something that can, in practice, be relied on?

My somewhat ignorant experience is full of dealing with combinatorial expansion of python build backends and needing to husband them in nixos including all sorts of overrides to deal with, iirc, weird circularities in deps.

You can demonstrate it somewhat empirically: pip, uv, etc. all understand the build backend interface and packaging standards and can install any package that conforms with the standards.

(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)

> any package that conforms with the standards.

Right, but what I'm asking is: by and large, can you expect any random package on pypi to conform to the standard?

Yeah, I think conformance is pretty good these days. PyPI itself enforces metadata well-formedness, and projects that don’t conform tend to get noticed quickly (because they don’t install correctly).

Reply on news.ycombinator.com