My hope is that conda goes away completely. I run an ML cluster and we have multi-gigabyte conda directories and researchers who can't reproduce anything because just touching an env breaks the world.
It's still very immature but if you have a mixture of languages (C, C++, Python, Rust, etc.) I highly recommend checking it out.
Perl was the language of the big hack. It had very little to offer in the way of abstractions and OO. So you were kind of excused from writing well structured code and people wrote a lot of janky, untestable stuff in perl and carried on doing than in python.
In python, to get good code you absolutely have to have unit tests and some kind of integration tests. I worked on a roughly 50,000 line python program. It wasn't bad even though we didn't have type hints then. Occasionally we did discover embarassing bugs that a statically typed language would not have permitted but we wrote a test and closed that door.
But day to day, I am happy to write Kotlin and Dart :D they give you amazing tooling and just good enough type guarantees.
Anyway, good to have a bunch of options :).
1. Installation & dependencies: Don't install Python directly, instead install pyenv, use pyenv to install python and pip, use pip to install venv, then use venv to install python dependencies. For any non-trivial project you have to be incredibly careful with dependency management, because breaking changes are extremely common.
2. Useless error messages: I cannot remember outside of trivial examples without external packages when the error message I got was actually directly pointing towards the issue in the code. To give a quick example (pointing back to the point above), I got the error message "ImportError: cannot import name 'ChatResponse' from 'cohere.types'". A quick google search reveals that this happens if a) the cohere API-Key isn't set in ENV or b) you use langchain-cohere 0.4.4 with cohere 5.x, since the two aren't compatible.
3. Undisciplined I/O in libraries: Another ML library I recently deployed has a log-to-file mode. Fair enough, should be disabled before k8s deployment no biggie. Well, the library still crashes because it checks if it has rwx-permissions on a dir it doesn't need.
4. Type conversions in C-interop: Admittedly I was also on the edge of my own capabilities when I dealt with these issues, but we had issues with large integers breaking when using numpy/pandas in between to do some transforms. It was a pain to fix, because Python makes it difficult to understand what's in a variable, and what happens when it leaves Python.
1. and 4. are mainly issues with people doing stuff in Python it wasn't really designed to do. Using Python as a scripting language or a thin (!) abstraction layer over C is where it really shines. 2. and 3. have more to do with the community, but it is compounded by bad language design.
2. I've used a ton of languages and frankly Python has the best tracebacks hands-down, it's not even close. It's not Python's fault a 3rd party library is throwing the wrong error.
3. Again, why is bad language design a library can do janky things with I/O?
4. FFI is tricky in general, but this sounds like primarily a "read the docs" problem. All of the major numeric acceleration libraries have fixed sized numbers, python itself uses a kind of bigint that can be any size. You have to stay in the arrays/tensors to get predictable behavior. This is literally python being "a thin abstraction layer over C."
2. I would argue that the ubiquity of needing stack traces in Python is the main problem. Why are errors propagating down so deep? In Rust I know I'm in trouble when I am looking at the stack trace. The language forces you to handle your errors, and while that can feel limiting, it makes writing correct and maintainable code much more likely. Python lets you be super optimistic about the assumptions of your context and data - which is fine for prototyping, but terrible for production.
3. I agree that this isn't directly a language design issue, but there's a reason I feel the pain in Python and not in Rust or Java. Dynamic typing means you don't know what side effects a library might have until runtime. But fundamentally it is a skill issue. When I deploy the code of Java, Go or Rust people, they generally know I/O is important, and spent the necessary time thinking about it. JS, Python or Ruby devs don't.
4. The issue is that Python's integer handling sets an expectation that numbers "just work," and then that expectation breaks at the FFI boundary. And once you're of the trodden path, things get really hard. The cognitive load of tracking which numeric type you're in at any moments sucks. I completely agree that this was a skill issue on my part, but I am quite sure that I would not have had that problem if it was in a properly type-set, compiled language.
Static typing (in most industrially popular languages) doesn't tell you anything about side effects, only expected inputs and return values.
Dynamic languages let you do a lot of things with meta classes and monkey-patching that allow you to use a library without needing to build your own special version of it. With some C or C++ library that did something bad (like the logging one you mentioned) there's nothing for it but to rebuild it. With Python you may well be able to monkey patch it and live to fight another day.
It is great when you're dealing with 3rd party things that you either don't have the source code for or where you cannot get them to accept a patch.
Now of course, we should have long removed the monkey-patch, or properly documented it somewhere, or I should have been more knowledgeable about Ruby and known that monkey-patching was a thing, but there's a reason why it is a pretty frowned upon practice.
I don’t understand why anybody would ever develop anything in Python other than “I want to write software but can’t be arsed to follow software design principles”, with all the mess that follows from it.
This is such a deeply unserious take. Do you have hundreds of thousands of hours to give out for free? No?
On the python front, however, I am somehow still an old faithful - poetry works just fine as far as I was every concerned. I do trust the collective wisdom that uv is great, but I just never found a good reason to try it.
It makes building FreeCAD pretty trivial, which is a huge deal considering FreeCAD’s really complex Python and non-python, cross-platform dependencies.
Want to make sure a software stack works well on a Cray with MPI+cuda+MKL, macOS, and ARM linux, with both C++ and Python libraries? It’s possible with conda-forge.
Arbitrary examples, I know, but I moved a large software that was truly mixed C++ and python project to conda-forge and all sorts of random C++ dependencies were in there, which drastically simplified distribution and drastically reduced compile time.
If I had done it today, it might be nix+bazel, or maybe conda+bazel, but maintaining a world of C++ libraries for distribution as wheels does not sound like fun - especially because nobody is doing that work as a community now
Conda is like Jquery or Bootstrap: It was necessarily before the official tools evolved. Now we don't need them any more, but they still are around for legacy reasons. You still need it for example, for some Moleular dynamics packages, but that's due to the package publishers choosing it.
It is probably the easiest way to install a lot of binary dependencies, good for who doesn't have experience with sofware development and don't care with reproductbility.
I wish my life had been like this. Unfortunately I always appear to end up needing to make this stuff work for everyone else (the curse of spending ten years on Linux, I suppose).
But then ML is a very broad church, and particularly if you're a researcher in a bigger company then I could see this being true for lots of people (again, i wish this was me).
- Enums with inner types require hacks
- No clean way to make your Rust enums into Python enums
- More boilerplate than you should have. I think I will have to write my own macros/helpers to solve. For example, you need getters and setters for each field. And the gotcha: You can't just macro them due to a PyO3 restriction.
- No way to use other Python-exposed rust libs as dependencies directly. You have to re-expose everything. So, if your rust B lib depends on rust A, you will make a PyO3 Rust A package, but instead of re-using that in PyO3 Rust B, you will copy+paste your PyO3 Rust A boilerplate into PyO3 Rust B.
https://docs.metaflow.org/scaling/dependencies https://outerbounds.com/blog/containerize-with-fast-bakery
does that fit the bill?
Conda packaging system and the registry is capable of understanding things like ABI and binary compatibility. It can resolve not only Python dependencies but the binary dependencies too. Think more like dnf, yum, apt but OS-agnostic including Windows.
As far as I know, (apart from blindly bundling wheels), neither PyPI nor Python packaging tools have the knowledge of ABIs or purely C/C++/Rust binary dependencies.
With Conda you can even use it to just have OS-agnostic C compiler toolchains, no Python or anything. I actually use Pixi for shipping an OS-agnostic libprotobuf version for my Rust programs. It is better than containers since you can directly interact with the OS like the Windows GUI and device drivers or Linux compositors. Conda binaries are native binaries.
Until PyPI and setuptools understand the binary intricacies, I don't think it will be able to fully replace Conda. This may mean that they need to have an epoch and API break in their packaging format and the registry.
uv, poetry etc. can be very useful when the binary dependencies are shallow and do not deeply integrate or you are simply happy living behind the Linux kernel and a container and distro binaries are fulfilling your needs.
When you need complex hierarchies of package versions where half of them are not compiled with your current version of the base image and you need to bootstrap half a distro (on all OS kernels too!), Conda is a lifesaver. There is nothing like it.
Conda ecosystem is forced to solve this problem to a point since ML libraries and their binary backends are terrible at keeping their binaries ABI-stable. Moreover different GPUs have different capabilities and support different versions of the GPGPU execution engines like CUDA. There is no easy way out without solving dependency hell.
It is also quite complex and demands huge investment of time to understand its language which isn't so nice to program in it.
The number of cached combinations of various ABI and dependency setting is small with Nix. This means you need source compilation of a considerable number of dependencies. Conda generally contains every library built with the last 3 minor releases of Python.
Like, I had real issues with GDAL and SQLite/spatialite on MacOS (easy on Linux) which uv was of no help with. I ended up re-architecting rather than go back to conda as uv is just that good (and it was early stage so wasn't that big of a deal).
The curmudgeon in me feels the need to point out that fast, lightweight software has always been possible, it's just becoming easier now with package managers.
I stayed for the native functional programming, first class enums, good parts of C++ and the ultimate memory safety.
Rust is for me similar to C just like you wrote, it is better, bigger but not the overwhelming way like C++ (and Rust has cargo, don't know if C++ has anything).