Rye: Flask author's new project for Python bootstrapping | Hacker News Reader

Rye: Flask author's new project for Python bootstrapping

github.com

208 pointsBiteCode_dev45 comments

Neat and it doesn’t fall into the same trap as many others trying to build similar tools in python itself by being built with rust there’s no bootstrapping issues with managing the python environment the tool to manage python environments needs.

However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. So while I’m impressed and think it looks good it’s a 100% no for me just because I have on multiple occasions had to compile ahead for testing, or apply a tiny patch python to build on an unsupported os such as a too old or new macOS…

I’m not the target audience but unfortunately I suspect sue to how well known he is, this will lead to adoption and popularity and while I think this tool is good for him and other people I don’t think python needs more opinionated management tool projects getting wildly popular… I don’t want to see a repeat of the pipenv saga

Author here.

> However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless.

For what it's worth I do agree with this, and in fact before I published this to GitHub, rye was capable of doing that. I somewhat intentionally stripped it down a bit to better explain and showcase what I wish that experience looked like.

In an ideal world, we would not have to use custom Python installations because a tool like rye could automatically fetch it. However there are always cases where you need (eg: local python builds for development etc.) so I will definitely want to bring this back.

The reason I never put this on GitHub is really that I do not want to build the next thing that just splits the ecosystem apart. But as of recent, there have been some developments that make me somewhat optimistic that there is a chance, Python packaging could be fixed once and for all, and this is supposed to be a bit of a teaser of what could be.

Armin, thanks for rye and all you've done throughout the years.

I'm curious, what are these developments you've mentioned that make you optimistic?

I think the general packaging situation has sufficiently improved so that the delta to a good outcome is quite narrow. This is from both the adoption of pyproject.toml and the build ecosystem around it as well as a desire towards portable python executables. (See also the existence of PEP 711)

This looks very nice.

I’ve been quite happy with Hatch, which I adopted about 3 months ago on the strength of the PyPA’s recommendation. I sort of like where they’re coming from philosophically - sticking as close as possible to standards - and find the API pretty nice. I’ve never used Poetry so couldn’t compare.

One concern I have is that it seems to largely be the work of a solo dev who also has other commitments. Absolutely nothing wrong with that in principle, of course, but I wonder how sustainable it is. I have just noticed that PyPA owns the repo - not sure if that was always the case or if it’s a recent development, but does bode well.

It wasn’t always the case. It used to lie at ofek/hatch

It's more than those tools as it provides python itself.

Has a similar look to the Posy/PyBi idea[0]. PyBi is also currently under consideration as PEP711[1].

[0] https://discuss.python.org/t/announce-pybi-and-posy/23021

[1] https://peps.python.org/pep-0711/

> No Core Proprietary Stuff: Rye (with the exception of it's own tool section in the pyproject.toml) uses standardized keys. That means it uses regular requirements as you would expect. It also does not use a proprietary lock file and use pip-tools behind the scenes.

The use of `pip-tools` to generate a `requirements.txt` as a lockfile is a big negative for me. Last I looked, they do not generate a cross-platform lockfile. In theory, you could still use `requirements.txt` for a cross-platform lockfile; it just might be a mess for doing incremental updates like when adding a new dependency.

How's support for PyTorch/Jax/Tensorflow-GPU? Poetry struggles when resolving cross platform bundled driver dependencies. So far only conda/mamba is able to handle it performantly.

If you can pip install those libs and they work on your platform then it looks like this should work--it's just a nice opinionated wrapper around pip, virtual env and other Python tooling. If you need conda installs because those libs have specially compiled native depencies (boatloads of fortran code, etc) then you're going to run into trouble and probably should stick with conda.

Another issue is that PyTorch installation requires the dependency resolver to scrape the dependency host's HTML like a RSS reader because pypi doesn't automatically store metadata.

https://github.com/python-poetry/poetry/issues/6409

Yeah anything that needs specially compiled or optimized binaries is an enormous pain in the python world. You really want conda for those cases.

The project is very new and unstable. Don't expect it to solve the whole thing today. But the concept is sound.

My hope is that the ideas on offer inform a PEP that gets into 3.13.

Python's packaging/project management story isn't among its selling points at the moment.

That's quite the understatement...

Mild >>> wild.

The workspace support looks really interesting, it's something that I'm really missing in Poetry right now.

Love the self-consciousness of it all, even having a "Should this exist" with an XKCD that jumped right to my mind as soon as I read the title.

Honestly, as someone who has to periodically jump into Python to do some scraping and data wrangling I dearly hope we land on something that is sane and maintainable in the near future. I'm just so tired of (ana)conda, poetry, pyenv, and even pip itself.

It is clever to use Rust so the tool won't break with each new Python release.

Isn't it also declaring complete bankruptcy as far as Python's own ecosystem is concerned, that you'd have to go to Rust to fix this?

Maybe if a tool like this is successful enough it could “bootstrap” the Python ecosystem into being as reliable :P

not really? It's not like CPython is written in Python either. Almost every language that has some notion of virtual environments writes a good chunk as shell scripts for maximum environment compatibility.

Sometimes you want to use the right tool for the job, and sometimes the scripting language that has a lot of environment-based configuration will not be a good fit for an "installer" that needs to handle a lot of environments cleanly.

I don't think that's right. Sure, the language runtime environment often uses a lower level language to speak with the underlying hardware or abstraction. But large parts of standard libraries may be written in the language itself. Build systems may have a little bit of shell scripting to get going, but larger such programs would often be written in the language itself. This certainly is the case with NPM, Yarn (JavaScript), Maven (Java), Gradle (multiple JVM languages), Rake (Ruby).

I think Ruby's equivalent would be something like rbenv (which is just a big shell script!). Java similiarly ends up often being installed through package managers or other bespoke system installers that would at the very least need to bundle a runtime.

To your point, One could write something like rye in Python, and then use PyOxidizer to package it up. I imagine a part of this is "Rust is fun", which is totally a legitimate reason to work it. And of course Rust comes with a very good distribution story (Go might have been another good answer for some people).

A good example for all of these is actually asdf. Which hilariously handles _every_ language just fine, _except for _ Python and Node. It is a terrible reflection on the state of tooling in these communities.

How does asdf not handle Python and node well?

For python it routinely has trouble with pip, default locations that pip use, virtual envs break, reshimming doesn't work half the time, I can't even remember all the other issues I've had.

For node it's more of the same, old versions of node were so broken that installing them through asdf doesn't work unless you build directly from source. Reshimming doesn't work half the time, global packages can't be found, idk what else. tons. It's just ridiculous to see when _every_ other language works fine in asdf.

And to add to your point: if even rust is preferable to python to write those kind of script-like tools, what does it says about the future of python ? (Yes, i know, ML, etc)

Not quite, as noted in other comments compiling the tool to a self-contained and unchangeable executable is a very effective way to avoid bootstrapping issues where to set up a Python environment (to do something useful) you need to setup a Python environment first (to run Python tools).

It's not like regular tools need to break with each new Python realease. It's mostly endless tinkering, attention deficit, and bad choices.

Heck, even within the same Python release, I never understood why we need to be in the 2000+ update of pip, for example, and it needs to update itself almost every month or so. It's a ducking dependency installer, just formalize the design and protocols already and keep it at that.

(For context, I never had that issue with npm. Not to any same-order-of-magnitude comparable level at least).

Python/pip criticism is downvoted again. The Python propaganda arm is strong as ever. No other language relies on marketing and censorship that heavily.

I encountered problems with dependencies in python projects only when AI/ML libraries were used pytorch/scikit-learn/etc

NPM on the other hand: it broke in every project and in trivial cases. E.g. upgrade of postgresql lib.

I hope that deno/smth other will make possible to forget npm finally

It's more the fault of broken libraries (remember that you depend on the weakest link out of hundreds or thousands of random strangers) than of inadequate tools.

I wish it supported setup.cfg

Out of interest... why? From what I can tell the community really seems to be adopting pyproject.toml as the standard and most tools are supporting it as their primary or only configuration option now. A few are using setup.cfg, but that seems to be for legacy reasons rather than an active decision. I'm not a huge fan of TOML as a configuration format personally, but it's at least more capable than cfg files. I'm really keen to know why you feel cfg support would be useful.

To be honest, I feel like setup.cfg hasn’t been given a chance to prove itself useful. Just as it was made possible to use setup.cfg only (without setup.py), it’s been almost immediately taken over by pyproject.toml. However, for those who actually migrated their projects to setup.cfg, migrating once again doesn’t bring that many benefits (but e.g. loses compat with flake8).

Besides, setup.cfg’s syntax is slightly more elegant, or at least I think so :)

By the way, why do you think it is more capable? As far as I know, setup.cfg has some feature toml (intentionally) doesn’t support (not that they’re useful in this context anyway).

I've always thought of TOML as not far off a superset of the configparser format, although with a different API in code. I realise most tools don't use the extent of configparser, but coming from JSON and YAML, I always found the more advanced configparser features counter-intuitive and awkward to use. I'm not a huge fan of TOML, but it has the necessary primitive values (strings, numbers, booleans), and some reasonable structures for managing collections and nesting.

Is there seriously another standard being set up in the Python community? When will it end? That’s not a rhetorical question, I’m seriously asking when the community is going to actually solve the tooling problem without making everything worse with each successive update.

rye's author was in before that https://github.com/mitsuhiko/rye/discussions/6

I'm not asking rye's author. I'm asking here. When will python maintainers decide that enough is enough and actually design a working language, tooling and all. Python is my _most hated language_ and it is because of stuff like this. I might actually use python or at least not hate it if people would stop pretending like it's the greatest language on earth and actually fix the worst part of it. If stuff like this keeps happening though, I seriously doubt that will ever happen.

Using a highly opinionated package manager with barely any tests? Supply chain attacks exploiting package managers are actually a thing to be extremely scared of. https://news.ycombinator.com/item?id=34836275

Off topic: it’s too bad Armin is doing so much in Rust, instead of Nim.

Would be great to have a recognizable Python developer, using Nim.

The point is not to prop up Nim. I'm not sure why it would be desirable even, aside from "but I like Nim".

The point is to improve Python's ecosystem. If they do it using a more established language than Nim, with great traction, that's for the better.

Aside from a very subjective "I like Nim to succeed" there is literally no rational reason to pick one over the other (except, maybe the other way around, because Rust has wider adoption).

So communicating that thought as if it was a generally accepted/objective statement on Armin's project isn't going to give you a lot of love.

Reply on news.ycombinator.com