Comment by pdonis | Hacker News Reader

Comment by pdonis | Hacker News Reader

Parent Full thread

> The nice thing about Python packaging is that it’s standardized

But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.

Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.

> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.

As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.

> I don’t understand why Gentoo needs to rebuild every single release of uv-build.

Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.

> you can do a normal version bound in your Python build backend metadata

Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".

distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.

Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again.

So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.

Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.

i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time.

my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.

for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...)

This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?

> i guess gentoo is rolling-release

That's my understanding, yes.

> my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.

Thing is, that apparently won't work. From [0], with some more discussion starting at [1]

> > I don’t understand why Gentoo needs to rebuild every single release of uv-build.

> Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.

[0] <https://news.ycombinator.com/item?id=50077466>

[1] <https://news.ycombinator.com/item?id=50077553>

Per my response, that isn’t how Python packaging works. There is no “use the latest version” mechanism in standard Python packaging; leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.

> ...leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.

So, would it be correct to decide that package with an unconstrained version gets built with 0.1.0? And when that fails to build, what's the distro packager to do? Bang on the package maintainer until they specify the real constraint? Yeah, good luck with that.

Doesn't it make way more sense to assume that "unconstrained uv-build version" means "they only tested this with the very latest version of uv-build" and proceed accordingly? This will fail sometimes, but I figure it'll fail far less often then assuming that you get to use 0.1.0 if no version is specified.

I'm being a little extreme with my example here, but only to illustrate the larger point I'm making. Breaking changes do happen, and without specified version constraints one has to assume that "very latest available" is the correct one.

Well, that’s why you should use a semver range, as is the default, instead of leaving it unconstrained. That’s the actual solution that solves the problems for all parties here; I’m just pointing out that “unconstrained = latest” isn’t a correct summary of how Python packaging works.

In that case, wouldn't it be better for uv to reject unconstrained? Gentoo (or any packager, such as homebrew / nix) is too far downstream for that. Presumably allowing unconstrained seemed to make sense in some circumstances at some point?

No, because it’s a standard part of Python packaging, and it’s not the problem here.

(There are all kinds of legitimate reasons to use an unconstrained dependency, including locking it separately and also trying to help other downstream Linux distros with their own bespoke packaging problems.)

you literally just said

  > that’s why you should use a semver range, as is the default, instead of leaving it unconstrained

You should probably pick one of these two mutually exclusive options and stick with it.

> Gentoo can't second guess you

gentoo appears to support exactly 7 versions of uv-build, a list they update by hand: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...

git changelog of uv-build in the gentoo package manager: https://gitweb.gentoo.org/repo/gentoo.git/log/dev-python/uv-...

gentoo also wrap python packages in their own package manager, packages in this tree appear to come from their upstream version control system eg Github, not PyPI.

just because someone on hn said something, doesn't make it true.

> they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection

I don't follow. This feels like Gentoo is mishandling build-tool dependencies.

Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?

Strictly speaking, they should, right? Differences in GCC could produce differences in anything it compiles.

IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed.

No, and not with clang or llvm or rust. When we get a new Golang tho, that does rebuild stuff built with Go. And big perl updates cause a rebuild for perl packages.

> I don't follow.

I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.

Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.

> This feels like Gentoo is mishandling build-tool dependencies.

I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.

> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?

No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.

> The build backend is an explicit part of the specification of a Python project

Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update - that's a behaviour they've decided is correct on their own.

> C/C++ libraries don't have an explicit specification of the build tool to be used in their project

They don't have a standard way to specify such, but they have an implicit dependency on the installed compiler toolchain, and it would be just as valid to interpret "no way to specify a build-system dependency" as "always rebuild" when the build tool updates

> Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update

Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update but nothing else in the project does. So what's a "build everything from source" distro to do? If it's possible that updating the build dependency will change the build artifact, doesn't it have to rebuild?

> Nothing in the Python spec guarantees that the resulting build artifact will be identical if the build system dependencies update

With this framing, exactly the same thing can be said about libstdc++, for example, so must we also force a full system recompile on every GCC/clang version bump?

Seems like a design flaw in gentoo that previously hadn't been an issue because build tools weren't being updated nearly as often as uv-build is.

I linked to some of the relevant standards below.

At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.

(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)

> it seems extremely unlikely that they actually need to build every single patch version of uv.

If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.

> Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.

Sure it does. TFA claims

> all Gentoo users need to build it from source, and given that it’s released quite frequently (12 releases in the 30 days prior to writing this post), that’s a lot of building. Like, for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.

And that is false, exactly because of that standard. If you want reproducible builds then you can stick with a previously built version of uv-build for a long time. If you want working builds on the fly using your own processes, then the standard empowers you to substitute something else. If you want to use the Python packaging ecosystem in the normal way, then it doesn't even matter what your distro is.

uv-build is for pure Python projects, which entails that the "build" process is trivial. It doesn't even bytecode-compile the .pyc files generally; wheels aren't expected to contain them (your installer may produce them at install time, or you can let Python produce them on demand). Binary reproducibility for pure-Python wheels is just not, from anything I've seen, a high priority for Python packaging.

Linux distros have these ideas about packaging that are just not suitable for Python and not in tune with what the Python ecosystem is doing. The vast majority of the Python code you install should not be integrated into your "system". (Python devs rarely find themselves wanting to "bootstrap" everything from source, either.) Gentoo's issue here appears to be their own policy, which is a problem of their own making. It's unrealistic for them to expect random Python devs to care about this.

> As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.

If you mean because of Gentoo policy, that's their fault. I can guarantee you that approximately nobody using uv-build is concerned with whether their package makes it into Portage. Python build systems aren't designed to produce packages native to Portage (or any other Linux packaging system). Probably the overwhelming majority of these users aren't expecting any Linux distro repositories to host their package in any form, and probably most of them couldn't even tell you what Gentoo is.

If you mean technically, then no; once you have the source code and are using ebuild, nothing compels you to choose the backend specified in pyproject.toml. If everything is up to standard then it's trivial to switch it over to, say, Flit. Most projects following a standard src layout will need no tool-specific config, and there's quite little of it possible regardless (and all well documented). The author even highlights:

> It’s a backend and the whole point of PEP 517 is that you can use any backend with any tool. You can use flit-core, you can use hatchling, and either of them will work just fine with uv (or any other frontend).

The edits do take time but it's a lot better than the proposed bogeyman of waiting for a new build of uv-build from Rust source.

> nothing compels you to choose the backend specified in pyproject.toml

This comes as a complete surprise to me. I thought the whole point of specifying the build backend in pyproject.toml was so the project owner can choose which build backend their project gets built with, and have that choice automatically enforced by tools.

Are you saying this kind of thing is allowed by the Python spec? If so, where? Can you give a specific document?

> The author even highlights

What he means is that the project owner doesn't have to, for example, use uv-build as their build backend in pyproject.toml just because they are using uv as their overall project management tool. I do not think he means that he, as a distro packager, is free to just ignore what's in pyproject.toml.

> the project owner can choose which build backend their project gets built with

When following an automated workflow implemented by an installer or other tooling, yes.

But you own the sdist file, and the archive contents, after downloading it. Nothing prevents you from editing pyproject.toml, using a build frontend/system that offers a flag to disregard PEP 517, running your own custom scripts on the data, manually copying the files into the target environment, or whatever else you can think of.

And since Gentoo is doing its own packaging with its own package format, nothing prevents them from doing the same kind of patching.

> you own the sdist file, and the archive contents, after downloading it.

If you're forking the project, sure. But I don't think that's what Gentoo is doing.

If you're trying to build what the project owner intended for you to build, I don't see how you can just ignore what the project owner specified. Nor do I see how that is allowed by the Python specs. That's why I asked you for a reference if you think it is in fact allowed by the Python specs.

If your position is that the Python specs don't allow you to just ignore pyproject.toml, but a distro can just ignore what's in the Python specs, I don't agree with the latter either, but it's a different position, and I'm not sure which one you're taking.

> uv-build is for pure Python projects, which entails that the "build" process is trivial.

The build process is more than just the Python code. There is also project metadata, which might be dynamically generated by the build backend. Indeed that's one of the main features offered by the different build backends, simplifying the project owner's life by removing the need to write a lot of boilerplate metadata that the tool can automatically discover on its own.

Yes, but these things are largely trivial in most cases.

In a wheel, there are basically two kinds of metadata: the kind that follow directly from the circumstances of the build (RECORD, which lists sizes and hashes for included files, and WHEEL, which says what environments the wheel is for and what tool built it), and then the kind that has to be specified by the developer. The latter is in METADATA. But the thing is, for most projects this data can be completely static, in which case the wheel METADATA is just a copy of the corresponding sdist PKG-INFO.

Build backends support a variety of dynamic metadata, but there are two places where that could be calculated: in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO) and when building a wheel from that sdist. The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.

> these things are largely trivial in most cases

The point is that that's not your call to make. It's the project owner's call. That's what the build backend specification in pyproject.toml is for. If you're going to just ignore that, what's the point of having it at all?

> in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO)

Which is the only place that's relevant for this discussion, since we're talking about Gentoo, which builds everything from source.

> The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.

I agree with all this, but as above, it's irrelevant to this discussion.

Reply on news.ycombinator.com