charliermarsh

Happy to report that this definitely isn't a paid ad. The Home Assistant team did this on their own. I helped out by building uv for some of the architectures they needed: https://github.com/astral-sh/uv/pull/2417

Thank you so much.

We do actually support direct installation of Git and direct URL dependencies. Like, you can do run `uv pip install` with:

``` black @ git+https://github.com/psf/black ```

We also support editable installs for local directories, like you mentioned:

``` black @ ../black ```

The thing we don't support is using _editable_ installs for Git and direct URL dependencies, like:

``` -e git+https://github.com/psf/black ```

In my experience, these are really rare, and we didn't see a reason to support them.

I think having a good story for local workspaces is gonna be a key piece of this next phase of the project, where we grow uv from a "pip replacement" to a "Poetry replacement" -- or, phrased differently, to something that looks more like Cargo. (Cargo does a bunch of nice things here that I want to learn from.)

Yes, we absolutely want to support platform-agnostic lockfiles. Very important thing.

The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).

Yeah, this is very much on our roadmap and probably one of the first things we'll tackle next.

Thanks Ed! Your work as always is a big source of inspiration.

> Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".

Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution.

> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.

This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.

This is how Ruff works too! (Ruff is also a standalone binary with no Python dependency.) If you're interested, I recommend checking out Maturin, which makes this pretty easy -- you can ship any standalone Rust binary as a Python package by zipping it into a wheel.

If you notice differences vs. the originating plugins, would definitely appreciate it if you could file an issue! We tend to be very responsive especially when it comes to matters of correctness.

Candidly, for Flake8 plugins, my experience is that Ruff tends to be more consistent, more robust in its inference and, at this point, more extensively tested than the original implementations -- both via our own testing and via the significant number of projects and companies that now use Ruff in production. (As compared to Pylint, though, we catch fewer issues, since Pylint does some type inference across files, which Ruff doesn't support yet.)

Ruff is also designed such that we will iteratively lint-and-fix until there are no more fixable issues, so if you've seen the linter introduce _new_ fixable diagnostics, that would be a bug too.

Regardless, thanks for giving it a try :)

Thank you for the kind words :)

Yeah that's right, we use the RustPython parser and lexer internally. Works great! And it's actually seen a bunch of improvements over the past few months -- we contributed back some CPython compatibility improvements and a couple new language features (parenthesized with statements, match statements, *except support).

By the way: epage, you are a hero. I am very grateful for everything you build.

Oh heh yeah I saw flake8-import-single :)

That might be equivalent to using Ruff's isort implementation with `force_single_line` but maybe that comes with other implications that you don't want for SQLAlchemy.

Thank you for the kind words, I hope Ruff's working well for you

Yeah Ruff and Pylint can be used in a complementary way. Pylint implements rules that Ruff does not, and vice versa. I think Ruff might actually support more _total_ rules than Pylint at this point (not a great metric), but Pylint does more cross-file analysis and type inference. [1]

Some teams use Ruff alongside Pylint, some have replaced Pylint with Ruff entirely (Dagster is a good example [2]), especially those already using a type checker alongside their linter.

[1] https://beta.ruff.rs/docs/faq/#how-does-ruff-compare-to-pyli...

[2] https://twitter.com/schrockn/status/1612615862904827904

Yeah we don't support arbitrary Flake8 plugins. We do plan to support plugins eventually but the exact architecture and API are TBD.

If there are plugins that would be particularly impactful for you, I'd be happy to take a look at adding them. (We support some of the plugins you use in SQLAlchemy, like flake8-docstrings and flake8-builtins, but not flake8-rst-docstrings which I'm guessing is a blocker.)

It's mostly the reasons you've hit on but I'll try to add some color to them based on my experience with Ruff.

1. The "fearless concurrency" that you get with Rust is a big one though. Ruff has a really simple parallelism model right now (each file is a separate task), but even that goes a long way. I always found Python's multi-processing to be really challenging -- hard to get right, but also, the performance characteristics are often confusing and unintuitive to me.

2. Ruff performs very few allocations, and Rust gives us the level of control to make that possible. (I'd like to perform even fewer...) We tokenize each file once, run some checks over that stream, parse it into an AST, run some checks over that AST, and with a few exceptions, the only allocations outside of that process are for the Violation structs themselves.

3. Related to the above (and this would be possible with CPython too), by shipping an integrated tool, we can consolidate a lot of work that would otherwise be duplicated in a more traditional setup. If you're using a bunch of disparate tools, and they all need a tokenized representation, or they all need the AST, then they're all going to repeat that work. With Ruff, we tokenize and parse once, and share that representation across the linter.

4. Again possible with CPython, but in Ruff, we take a lot of care to only do the "necessary" work on a given invocation. So if you have the isort rules enabled, we'll do the work necessary to sort your imports; but if you don't, we skip that step entirely. It sounds obvious, but we try to extend this "all the way down": so if you have a subset of the pycodestyle rules enabled, we'll avoid running any of the expensive regexes that would be required to power the ignored rules.

Didn't expect to see this here but I'll take this opportunity to say thank you to all the contributors and to anyone that's given Ruff a try.

I'm around if anyone has questions.

Of course. There's at least one Ruff contributor who I know is working on a Python docstring parser, written in Rust, with formal grammars etc. (unlike Pydocstyle which relies on regex). I don't know the status of that project, but it would be very cool to see, and Ruff could of course leverage it :)

That's right -- we don't support match statements, and we don't support except* (which is part of 3.11 IIRC), but I _think_ we support everything else. And of course, I intend to support all of those language features.

If interested: CPython moved to a new parser, a PEG parser, in Python 3.9, and part of the motivation was to support language features like pattern matching, which introduced ambiguities in the grammar that the existing parser couldn't handle. For the same reason, they've been non-trivial to implement in the RustPython parser -- they either require clever techniques, or the parser needs to be written as a PEG parser. I am hoping to do the former, but I need to find time to prioritize it.

I've never used, but have heard good things about darglint (although it's now archived): https://github.com/terrencepreilly/darglint

A lot of folks want us to add that kind of enforcement to Ruff (https://github.com/charliermarsh/ruff/issues/458), and I want to do it, but it's a big project and some other stuff has higher priority :)

There's an issue tracking parity here: https://github.com/charliermarsh/ruff/issues/970

Though it's probably a bit conservative. Some rules are implemented as straight one-to-one ports from Pylint, and those are easy to check off, but others are Pylint rules that we've already implemented under other names, and those have to be tabulated on a case-by-case basis.

> There are two tools for dealing with packages in the Node world, namely npm and Yarn.

Not to mention pnpm. Or Bun. Or even Deno. All of which deal with packages, and make different choices.

This isn't to be a pedant, but I think it's a reasonable example of why it's less important that there's a single way to do a thing, or even a single tool vs. multiple alternatives; instead, what I care about is that I don't have to string together tens of tools to do that thing, and that there's some degree of compatibility and standardization in the API between those different alternatives.

I agree!

Oh, also, I think I've read the entire Discussions thread on this PEP, and from what I can tell, the PEP is missing champions who are eager to push it forward.

Part of the problem, I'm guessing, is that the PEP kind of sits in a weird space, because it's not up to the PyPA to "approve" it, or whatever -- PEPs are approved by the Python Steering Council.

Per Brett Cannon (Steering Council Member):

> Either consensus has to be reached (which it hasn’t) and/or someone needs to send this to the SC to make a decision (although it is a weird PEP in that it also impacts packaging, so it isn’t even clear who would get final say).

https://discuss.python.org/t/pep-582-python-local-packages-d...

It's interesting to note that PEP 582 (i.e. __pypackages__) mode used to be the PDM default -- and was sort of its flagship feature -- before being made opt-in with 2.0 release, the explanation being that the PEP had stalled, and that editor and IDE support for virtualenvs is much better (https://www.pythonbynight.com/blog/using-pdm-for-your-next-p...).

If you read through the discussion on the Python forum, one of the counterarguments to the PEP is that "If existing tools can already support this, why do we need a PEP for it?" But presumedly a PEP would help push the ecosystem to accommodate __pypackages__, and to solve the aforementioned problems (like broader editor and IDE support).

For what it's worth (as someone that builds Python tooling full-time): I'm generally a fan of moving to a node_modules-like model.

Absolutely no idea if they’re true but the Pipe founders are contesting these claims in the replies:

https://twitter.com/joshmangel/status/1595872694549069827

https://twitter.com/zallarak/status/1595873700846440453

Another project I've been following in this space (no affiliation) is Tangram (https://www.tangram.dev/), which I think of as "Nix, but TypeScript" -- or, from their Discord:

> Tangram takes a lot of inspiration from nix, and could be described as a mix between nix and bazel where you write JS/TS

The difference to me is that Depot is giving you what is effectively a dedicated build machine. That machine can take advantage of local BuildKit caching.

With Cloud Build, you need to use —-cache-from and other amendments to leverage remote caching. (And I believe that caching multi-stage builds require you to explicitly bush and push every stage.)

With Depot, you should in theory get all of that caching “for free”. You can also take advantage of things like BuildKit cache mounts (local directories on the host machine to cache, e.g., pip install artifacts) which AFAIK can’t be done with Cloud Build.

(No affiliation with Depot but I have played around with it.)

Another unusual alternative is to model your test running _as_ a Docker build. It's what I ended up doing in Trellis, which supports Depot as a runner (https://github.com/charliermarsh/trellis): running the linter check is modeled as "build the image, then add a `cargo clippy` RUN command as the last layer".