Why Python Is Terrible
josvisser.substack.com
josvisser.substack.com
Oh and he just says what is supposed to be quiet part at the end:
>And, not to put too fine a point on it, but if you can code Python but not Go (or another decent programming language), you probably have no business writing software for a living. I know it sounds harsh and I apologize for that, but writing software is a profession and hence you should have the knowledge, skill, and experience to use professional tools.
Hear that all data scientists, flask devs, systems engineers, and ML folks? Python is bad so you should quit. ;)
This is the biggest problem in software and it's kind of intractable.
The ideal world has tools that empower everyone to do what they need to do, which to some extent must include an activity like programming.
But, and this may be unconscious, "people who program for a living" have a strong incentive to gatekeep.
Not only then is it gatekeeping, it's also a sign of an inexperienced programmer.
I'll give one thing to Python programmers, they tend to work directly with their end users and build solutions that matter to them. Anyone that's done that kind of work knows it's difficult, regardless of programming language being used.
I know that's a tough thing to say -- but yeah, the irony is that a really really good "low-level programmer" is destined for obscurity, because things just work and once they're working people forget about it. Shouts to Linus Torvalds.
The amount of idiotic … implications of his statement is so excruciating it’s physically painful to me. But I encounter this unadaptive unawareness all the time with working with programmers from other teams, etc so I’m used to it.
Interesting contradiction.
I know someone like that, very energetic, full of ideas, but so stubborn I'm sure it holds him back.
Yeah, this kind of hyperbolic headline article repeating mostly dead-horse arguments is just low-effort click farming, not a serious contribution to the discussion of anything. All it adds to what is a well-trodden debate is…factual errors, like the claim that Python prominently features “lazy evaluation” (while, as in any language, you can arrange laziness in Python, it is very much eager normally.)
The ecosystem is the real plus, of course. But the language is a headache for this. I agree with the "false economy" angle. I would happily trade the "agility" of dynamic "gluing" with some kind of real type safety, human-readable error messages and performance[0].
[0] - hiding C in Python's clothes doesn't count :)
Quoting Brooks (butchered): “the only significant productivity improvement comes from relying on code that is already written”. Your fancy “better” language has not even 1/10th of what python has, it won’t replace it.
I never contested that.
> ML is very specifically about manipulating data (a fundamentally dynamic task)
I disagree strongly with using dynamic languages for data. Data has dimensions, units, types. You need to know that you're not adding coats to horses or USD to EUR. You need to know that you didn't silently sliced by the wrong axis. You may want formal verifications. You may to transform data without worrying about silent errors.
All the "metadata" and wrapper classes ML in Python are just trying to give you what the language can't.
> a very glue-task. Give me any language better than python for that
That's my point exactly. ML has evolved beyond glueing a few C libraries. It needs complex, big programs, which is an area where Python is terrible. Also the different nature of the "glued" components (each with its own data formats, protocols and calling conventions) makes the glue a mish-mash of untyped mixed magic idioms.
Part of the profession of software engineering is maintaining software that's already written. Should the people who maintain python code, not be paid for their work?
Another part is choosing the right tool for the job. Python has its flaws, but it is better than Go in some ways. For example, it has a richer ecosystem of libraries.
Why hasn’t the go community, of professional software engineers built an even richer ecosystem of libraries?
Is it ennui, incompetence, or attitude?
As Go came from Google, is that the attitude was, “I am a professional I’ll just write my own code to solve X”, rather than considering building a library that others can use?
Are libraries harder to build in Go? Is adoption of libraries by the Go community different?
Is it a mindset, that libraries are uninteresting?
Or is it something else entirely?
Others have stated that they don’t need such libraries in Go, or that Python just has a glut of libraries that are a sort of detritus obsolete or language version specific.
Why isn’t Go then, THE language of choice for machine learning, analytics, fast prototyping, data conversion, ETL, etc.
What I’ve heard of Rust in comparison is that the syntax of the language & the learning curve is counterintuitive for engagement by non-computer scientists or systems programmers.
Seems expected then that, non comp-sci folks will choose a language that is more accessible, with an ecosystem that lets them get things done in a short period of time.
I would wonder if it it that the experts & heavy users of Go or language architects moving Go forward see any need for a parallel course of engagement to bring Pythonists into the fold or to allow Go to become the language people choose over Python or TypeScript, et al, by providing a best of class approach to quickly prototyping and extending into well architected systems as an underlying function.
The computer industry has a bizarre reputation for moving fast and breaking things. In fact the industry is shockingly conservative. You will encounter many, many programmers who flat-out refuse to learn new things.
Leaving Silicon Valley, I’ve found the engineering management culture can be averse to training and allowing active skill building via development projects.
Just code that shit in Java or C++, doesn’t matter if you haven’t been trained in SQL properly or are even aware of best practices.
That culture leads to some gruesome product implementations & upgrade scenarios.
Hell, our IT department blocks this very website from being accessed.
I access it from my personal devices.
Plus, I've been programming in Go professionally for a while and it's been a while since I reached for a critical library and it was missing.
Go ends up needing fewer libraries anyhow; in Python you have the pure-Python version, no, wait, the Twisted version (which may not be current but are still all there, increasing the library count), no, wait, the async version, no, wait, the Python 2 version, no, wait, the one that binds to a C library.... and actually this isn't specific to Go, it's really more specific to Python. Library count gets bloated up over the decades by the fact that when, for example, Python went to async, all non-async libraries in which async was relevant suddenly needed a clone. Go has had effectively no language changes which create such parallel libraries (note changes, not additions; Go has had additions which create opportunities for more libraries but don't create lots of parallel libraries). I don't know that Python is uniquely changing, "best practices C++" is arguably up there, Javascript has had a lot of churn, but it's in the higher tier of such things. Languages that don't churn like that should be able to cover the same amount of "need" for libraries in fewer libraries by the numbers.
Might it be that coding in C was and is hard and further that memory management pointer issues and more led to unstable code?
Why hasn't the Visual Basic for Office community produced more libraries?
Linting has saved my bacon more than once, granted.
Ocaml is an interesting example imho, as the type inference engine is so good you hardly have to specify types. When you read it and squint, it looks dynamic. It's not.
You acknowledge that the kinds of static analysis that are feasible in Python are valuable, but it's "terrible" to want the kinds of static analysis that are infeasible. How interesting that the two boundaries line up exactly.
If you want the freedom to easily mangle json or other dynamic types, you'd enjoy python.
What's left are people who don't need that performance, which is sometimes me and is when I still am happy to use Python for something, and people who do need that performance, but don't know it. Those are the ones who get into trouble.
I do wish that the Python developer community would be more open about the fact that Python performance is really quite bad in many cases and that while there are some tools to peck around the edges, it is still fundamentally a low performance language, things like NumPy notwithstanding (it is, ultimately, still a "peck around the edges" tool, even if the particular edge it pecks it does extremely well, but that only makes the performance delta worse when you fall out of its accelerated code path). I feel like maybe back in the 200Xs the community as a whole was more open to that. Now "Python is slow" is perceived by the community as an attack. Maybe because the ones who did understand what that issue was are mostly gone.
But in the 2020s, yes, Python ought to be straight-up eliminated from many tasks people want to do today on straight-up performance grounds. Overcoming a ~40x disadvantage in the single core era was sometimes tough, but often doable. But nowadays basically that gets multiplied by each core on the system, and overcoming a multi-hundred-factor disadvantage is just not worth your time IF that is a level of performance you need. Python is nice, but not nice enough to pay for multiple-hundred-factor performance penalties. Static languages have come a long way since 1995.
The semantics of Python are fundamentally slow. To fix that requires changing semantics. Such a change would dwarf 2 -> 3 in size and be effectively a new language, like "Perl 6" was.
"This wrench is really bad for hammering nails!"
It's just that this article isn't very good at it.
It honestly just strikes me as very odd how e.g. even "just use multiple languages" is done a whole lot, but not really talked about as a good idea (as much as MY LANGUAGE IS GOOD AND YOURS IS NOT)
I've seen both sides of it- I worked in the same sub-org as Jos, and my very first job was cleaning up a large pile of incredibly important and variable-quality Python code used to manage a fleet of database servers. the code was tools to do useful things like implement failover of the replication master (across ~90-120 shards) from one region to another for maintainence. Or apply schema changes across all those master shards. Or monitor the shards at runtime.
At times, the code would Exception (literally, crash with a Python exception) during the middle of an important but routine maintainence, and the migration would be half-done. I was hired- literally, this was my job- to add tests to the code until it was much more reliable. Working on that convinced me that rather than type-safety (which is nice, and can be used optionally in python), high test quality and high test coverage of paths used in production were more important to keep the code running smoothly.
I just wish Python hadn't made strings a sequence type, as one of the most common errors at the org was accidentally e m a i l i n g e v e r y s i n g l e letter in a To: string. IE, if it was "To: bore-sre@stoogle.com", then b@, o@, r@, e@, etc, would all get an email saying "Process failed..." And r@ would reply (because he's rob pike) saying "your python program has a bug...."
> The problem with Python is of course that it is an interpreted language with lazy evaluation
That isn't "the problem" with Python. There's nothing wrong with these sorts of languages. However, it does limit the sorts of problems the language is suited for, and I do see places where Python was used where another language would have produced far better results.
Perhaps using Python inappropriately leads to some thinking that the fault is with the language?
I know it has functions that are lazy, but it's not lazy as in a sense that Haskell is right? I never use it as I find it a ghastly horror show (my taste, other people like it, that's fine), but I had to use it a few times and found that (also from the article) some parts are lazy, but not python as a language. Is that not correct?
> it does limit the sorts of problems the language is suited for,
Interpreted (...) is the implementation, there is no reason why it should be interpreted.
Nobody uses "lazy" in the sense of TFA.
It doesn't bode well for a bit of criticism when it starts by grossly misusing a technical term.
It's the less insane explanation. The alternative, that the author read and understood what lazy evaluation is, but somehow still thinks Python does it this way, is too crazy to consider.
Iteration in a sense can be "lazy", but that laziness is via data structures built on top of a strict core language. Python is not alone in having lazy iteratable data structures, but it leans into them relatively hard in its standard library.
Many of us love this, as it lets for loops work elegantly over all sorts of abstractions, but I could see some folks disliking it. Still, the author does not explain this nuance, which makes me think he does not quite know what he is talking about.
In an odd twist, function argument defaults are evaluated at function definition execution time rather than call-time (so half-way in laziness) so a `[]` as a default is shared between all users of that function and if you modify it you are changing a global. I heard that this led to a vulnerability within some security software.
Alternatively, maybe the author (and you) meant "interpreted language"? Lazy/strict evaluation is an orthogonal aspect.
https://blog.codinghorror.com/exception-driven-development/a...
https://stackoverflow.com/questions/2184935/performance-cost...
f, err := fn(...)pattern. Which is better is really a matter of taste. To have error handling smattered all over the code seems more inelegant and messy to me.
Personally, from the perspective of a number parsing function, I think being passed an unparseable number counts as an "exceptional" situation- InvalidArgumentError- and I don't care which way it's returned- as an raised exception, or as an error object- as long as it's clearly documented, and the use matches the semantics of the function (NetworkNotAvailable is a better example of an exception where you're want to put something in the exception block)
Eric Lippert had a decent blog post[1] back in 2008 on this titled "Vexing Exceptions" with regards to .NET.
What do you expect it to do? Silently fail and move on?
I write a lot of Go and I’m used to that. But when I hit snags in the less familiar territory of geographic data processing, it was a slog. Terrible documentation for the libraries was a major barrier, and otherwise it seemed as though essential features simply didn’t exist and I’d have to invent them.
I got the idea to explore Python for the project because people use it for data processing. I’ve used it in the past, though never for this. Whatever, I thought, at the very least I can validate that Go is a suitable tool.
Within a day I had rebuilt everything with Python. I built a flask app around the forecasting and fire perimeter tools, and had it deployed the same evening. It was mind blowing.
As an ecosystem I was absolutely blown away by Python. Do I like the language? Not really. I encountered so many cases where something could be so much faster and more efficient with Go. Deployment would be easier. I’d get more API from the same resources. Scaling would be ten times easier. Static typing tools kept blowing up my IDE because this library doesn’t support this, or the type tool is wrong about that. It was very janky at times.
Yet Python got it done. It’s live right now, and development is steady. Go was not steady and I didn’t see any solution to that in sight without reinventing countless wheels.
Months or years later, it's a beast, hard to understand or refactor, full of holes and pitfalls, and Python's terrible tooling doesn't help either.
And I never learn the lesson!
The linter is ok, and an occasional lifesaver (but it shouldn't even be needed! It requires extra work to catch problems that other languages catch "for free"). And it shouldn't be a separate tool. It's also cumbersome to use, silence what is not needed (way too noisy) and fine-tune it. Inline comments to enable/disable it for specific warnings look ugly, too. Python devs tend to suppress whatever bothers them instead of fixing it because it's not in their culture.
The type hinting checking is terrible. It's getting better, but it still misses obvious things and requires too much hand-holding. In my experience, average Python devs don't use it because they don't understand it, or don't find the ROI worthwhile. And because it's optional, they can just pretend it doesn't exist (or complain if you make it mandatory).
The mess that is dependency management has been discussed multiple times. In Python's defense, it's in "good" company with other messes from different languages. But Python's case seems particularly horrifying.
In general, with tooling, Python lives in a special hell where every blog and article will tell you "it's awful because you're doing it wrong, you should instead [use|avoid] pip, pyenv, pipenv, poetry, <my custom script>, <some deprecated tool that nobody else recommends>, <cutting edge tool that is incompatible with everything else>".
> And, not to put too fine a point on it, but if you can code Python but not Go (or another decent programming language), you probably have no business writing software for a living.
Oracle started buying Sun (and consequently Java) in 2009; the merger was completed in 2010. So I fail to see how that could have influenced Google's purported aversion to Java in the early 2000s, even as late as 2007 or 2008.
Python (fronting C++ code) still plays a huge role at Google. I don't see that changing. Go has almost zero story for scientific computing.
Come on man. There's being opionated, and then there's this.
The problem with Python is of course that it is an interpreted language with lazy evaluation […]
Far from me to defend Python, but this rant didn't start well.
I agree with the author that there are better languages for large applications.
If you expect to find Java or C in Python you are looking at the wrong place.
I'm glad you were able to port your scripts, but python is absolutely not appropriate for mission critical software.
> > Unfortunately these workers ran out of memory quickly so we decided to have workers terminate themselves after a configurable number of requests had been handled so that Linux would do our memory management for us.
I once worked with a service in Python that was essentially an image server for a special biomedical image format. The decoder was a proprietary C library written by a vendor and, unsurprisingly, had tons of memory leak issues. This exact feature of Gunicorn [0] saved us months of going back and forth with the vendor to get the library to stop killing our servers.
Python has it's flaws, but so does anything touched by humans.
[0] https://docs.gunicorn.org/en/stable/settings.html#max-reques...
It's extremely frustrating that you're forced into using it to access technology that doesn't even use Python really it's just the composing glue sticking the native C++ or GPGPU code blobs together.
I like this comparison. Anyway, it's interesting to note that it took the author "many years of experience running large applications written in Python" to come to his conclusions. The advantages of static typing and the disadvantages of dynamic or duck typing are well known since decades. The problem is less Python as a language, but the fly-by-night decisions to just use it for anything. To stick with the example: what prevents people from using "Lego bricks" (or a high-temperature proof version thereof) to build a reactor? Sound engineering decisions and, most importantly, safety regulations.
File this one under "things that UNIX systems programmers think will work in principle but end up being massive black-holes of attempting to quiesce any non-trivial application in a way that results in a sensible post-fork state".
Python is a horrible language because it is not a language. It is a family of languages that changes every October. Sure, 3.x doesn't introduce as many backwards-incompatible changes per version as 2.x did, but that's like saying World War I wasn't horrible because towards the end fewer people seemed to be dying every week.
If you have a couple scripts, sure, maybe you're not affected. But when you buy a company that shat out 90kloc of python and then all the employees quit, it's not a happy day.
And sure... I shouldn't have used those features. I get it. I'm the one who's bad because I'm calling your baby ugly. Even though I wasn't the one that originally wrote that code.
Though I did write some code that used package variables. And then the syntax for package variables changed, but that was an easy fix. And then the scope of package variables changed to be class variables, which is totally fine, but harder to find. Then the syntax changed again, but in a way that made it harder to fine. And then the debugger stopped working if you enabled async io for a few versions.
Python is for total amateur code fluffers.
Until it introduced the haphazard type system. Now I need to import types in every file, use IF to guard it in CI in every file, and use a powerful IDE to be able to use the benefits of typing.
I still use python. The recently introduced match statement is a great addition, IMO.
You don't need to do anything, you can ignore all type hints
> use IF to guard it in CI in every file
Are you talking about "if TYPE_CHECKING:"?
Your other option is to put "from __future__ import annotations" at the top of the file, or wait for Python 3.13 when PEP 649 lands and type annotations become lazily evaluated.
> Python's use of reference counting defeated copy-on-write because even memory blocks holding variables that were read-only were actually written to in order to manipulate the reference counts, thereby blowing up the combined physical memory footprint of the workers. We solved this by smurfing the interpreter to use a magic reference count number for all variables that were created by the master process and inherited by the workers, and then not touching reference counts that had the magic value.
Thanks
Those modifications force pages which were created on forking a child process as copy-on-write (meaning they share the same physical page until the page is modified by the child) to be copied and thus blow out any memory savings that would normally happen with copy-on-write.
This would be a nonstarter if it required N+1 times the memory of the single process, so the OS uses an optimization called copy-on-write. When a process forks, all its physical memory is shared by the new process so it takes almost no new memory to start. If the new process writes to a memory page, that physical page is copied so it has its own version. (Thus “copy on write”.)
For most programs this works fine, but if you have a runtime that does garbage collection using a technique that requires writing to an object even if the code doesn’t change any of its values, trouble ensues. With reference counting, you have to write a new reference count for an object anytime a pointer to the object is assigned. If you store the reference count in the object, that means its physical page has to be copied. So now the CoW optimization totally doesn’t work, because just referencing an object causes it to take up additional new memory.
Ruby used to have this same problem, and after Ruby webservers became popular (hello Rails) they eventually incorporated a patch to move the GC information somewhere outside the actual object heap. Other systems like the JVM use similar techniques to store the bookkeeping bits somewhere other than the object field bits.
So what the OP did is patch the runtime so the objects created in the master process (pre-forking) have special reference counts that are never altered. This mostly works, because the master process generally does a bunch of setup so its objects were mostly not going to be garbage anyway.
My problem with python is its package system, and the mess around the fact it was designed to be global. (I have a similar gripe with Ruby).
Ummm... okay.
I'm not going to cheerlead for Python here (in fact I do not like it at all and also avoid it whenever possible) but many of this author's points seem to boil down down to "screwdrivers are bad, here's why you should always use hammers instead".
Different tools exist for different purposes.
Lack of static typing is nothing in comparison with lack of common sense and unwillingness to learn ("why force oneself ? The job market swallows everything anyway") .
I have never found any language community to lack that, and if anything its more where people have lots of experience exclusively in one language than with inexperienced programmers picking up their first (who tend to, by nature, have a willingness to learn, even if they have a lack of the common base of experience that gets labelled “common sense”.)
Whenever some project I find doesn't just work, or works and then a few weeks later stops working, it always seems to be python. People cannot semver properly (even python itself? or is that some programs work on 3.9 and not 3.10 the programmers fault again?) and also cannot lock their dependency versions properly. Same problem that can happen to nodejs code, and yet, I rarely see such basic failures there.
I also just don't understand why anyone would even want to use python, anyway. I've tried to debug python code, or write new python code, but I could never get into it, nor was it easy to read code others had written. Whitespace sensitivity? Special __ files for package purposes? No JIT (no idea why pypy is a separate thing rather than rolled in)? I just don't see the advantage over JS which seems to do everything better. It even has the benefit of being syntactically intuitive for users of other languages, and typescript really has saved JS from chaos. It's fine to be different from C syntactically, but it I don't see the benefit of python to justify that syntax taking up another spot in my brain I could use for learning rust or something.
Pytorch is a great library too. It is hard to imagine Python decreasing in usage any time soon.
I'm glad I never reported to him while at Google.
The post is basically a surgeon complaining that a chef's knife is terrible for surgery; or how the F-16 can't loiter in the battlefield and deliver strafing fire; or how carbon fibre tubes don't handle compressive strength under water; or using C to do string manipulation and complex regex.
You're using the wrong tool for the job.
This is the only relevant statement.
The future is artificial intelligence programming Python and human programmers writing blog posts about how terrible Python is.