Guido van Rossum joins Microsoft | Hacker News Reader

Guido van Rossum joins Microsoft

twitter.com

1,337 points0xmohit799 comments

So instead of BDFL for Python, he's going "make using Python better".

Congrats to him for finding something fun to do in retirement - dictators usually end up with a different outcome. ;)

I'm looking forward to seeing the future of Python - I think this move will be great for the whole community, and lets him push boundaries without being bogged down on the management side.

An official package manager with great dependency resolution would be fantastic. Or over take pipenv or poetry and sponsor it through Microsoft $$$.

The biggest hurdle to python right now is the stupid package managers. We need cargo for Python.

poetry feels like the closest equivalent to cargo that I've used. pipenv is better than the previous status quo but is still oddly unstable, with random new issues I encounter with every release. poetry "just works" for me, has better dependency resolution, and IMO has a nicer interface and terminal output to boot.

Could you elaborate on what issues you've had with pipenv? I've only had very good experiences with it, so I'm surprised how many people here seem to prefer poetry.

https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...

Regarding not being able to work outside the project root[0]: This is actually one of the things that I love about pipenv! Anaconda, for instance, has environments that are not tied to a directory and are referred to by name (rather than by a directory path) and I've found this to be an absolute nightmare and extremely cumbersome! Not everyone has 10 projects that can share the same environment. (Besides, I would argue they never should.) I, for instance, have 10 projects that all require a slightly different environment and it's much easier to type a generic `pipenv shell` on the command line no matter what project I'm in, rather than trying to remember the Conda environment's name time and again. (Besides, it can be easily automated using .bashrc.)

[0]: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...

I only know a bit about Python - in what sense is pip not a package manager?

in the functioning sense.

care to elaborate?

> in what sense is pip not a package manager?

It is a package manager but it lacks features that many other package managers have in Ruby, Node, Elixir, and other languages.

For example there's no concept of a separate lock file with pip.

Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies.

The frozen file is good to replicate versions across builds but it's really bad for human readability.

Ideally we should have a file made for humans to define their top level dependencies (with version pinning support) and a lock file that has every dependency with exact pinned versions.

Isn't that effectively the result of a typical `setup.py -> pip compile -> requirements.txt` flow?

The setup.py file contains a human readable designation of requirements and then `pip compile` generates a requirements.txt with all deps' (and deps of deps') versions specified.

> Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies.

Isn't that a good thing?

> no concept of a separate lock file with pip.

setup.py/.cfg vs requirements.txt, no?

yes, but don't underestimate the power of convention.

if you make pip run 'pip freeze > requirements.txt.lock' after every 'pip install whatever', you almost solve that particular problem if setup.py is configured to parse that (it isn't by default and there's no easy way to do that!)

That's the whole point of distinguishing between logical dependencies and reproducibility dependencies. I use setup.cfg to describe the logical dependencies, I supply a requirements.txt (or environment.yml, or a Dockerfile) to provide the tools necessary to create a deployable build.

> Isn't that a good thing?

Yes, a very good thing.

> setup.py/.cfg vs requirements.txt, no?

A lot of web applications aren't proper packages in the sense that you pip install them.

They end up being applications you run inside of a Python interpreter that happen to have dependencies and you kick things off by running a web app server like gunicorn or uwsgi.

For a Python analogy vs what other languages do, you would end up having a requirements.txt file with your top level dependencies and when you run a pip install, it would auto-generate a separate requirements.lock file with all deps pinned to their exact versions. Then you'd commit both files to version control, but you would only ever modify your requirements.txt by hand. If a lock file is present that gets used during a pip install, otherwise it would use your requirements.txt file.

The above work flow is how Ruby, Elixir and Node's package managers operate out of the box. It seems to work pretty well in practice for ensuring your top level deps are readable and your builds are deterministic.

Currently there's no sane way to replicate that behavior using pip. That's partly why other Python package managers have come into existence over the years.

I don't understand the distinction you're making. Are you pip-installing or not? If not, why not?

My method for deploying a web application is to have a Dockerfile which pip-installs the Python package, but I could see someone using a Makefile to pip-install from requirements.txt instead. In fact, I use `make` to run the commands in my Dockerfile.

> Are you pip-installing or not? If not, why not?

I am running a pip install -r requirements.txt when I do install new dependencies. I happen to be using Docker too, but I don't think that matters much in the end.

Docker does matter, because the Docker image should take the place of requirements.txt (your "locked" dependencies) in your deployment process. I suggest you pip-install the package, rather than the package's requirements.txt file.

> Docker does matter, because the Docker image should take the place of requirements.txt (your "locked" dependencies) in your deployment process.

In practice it doesn't tho.

Let's say I'm working on a project without a lock file and commit a change that updates my dependencies. I get distracted by anything and don't push the code for a few hours.

I come back and push the code. CI picks it up and runs a docker-compose build and pushes the image to a container registry, then my production server pulls that image.

With this work flow there's no guarantee that I'm going to get the same dependencies of dependencies in dev vs prod, even with using Docker. During those few hours before I pushed, a dep of a dep could have been updated so now CI is different than dev. Tests will hopefully ensure the app doesn't break because of that, but ultimately it boils down to not being able to depend on version guarantees with Docker alone.

There's also the issue of having multiple developers. Without a lock file, dev A and B could end up with having different local dependency versions when they build their own copy of the image.

I've seen these types of issues happen all the time with Flask development. For example Flask doesn't restrict Werkzeug versions, so you wake up one day and rebuild your image locally because you changed an unrelated dependency and suddenly your app breaks because you had Werkzeug 0.9.x but 1.x was released and you forgot to define and pin Werkzeug in your requirements.txt because you assumed Flask would have. The same can be said with SQLAlchemy because it's easy to forget to define and pin that because you brought in and pinned Flask-SQLAlchemy but Flask-SQLAlchemy doesn't restrict SQLAlchemy versions.

Long story short, a lock file is super important with or without Docker.

Use the same method to verify in dev as in staging (Docker image). If you don't know it works in staging, then you didn't know in dev either.

FWIW I had a lot of success using https://github.com/jazzband/pip-tools to have dependencies automatically managed in a virtualenv.

* Basically I would have a single bash script that every `.py` entrypoint links to.

* Beside that symlink is a `requirements.in` file that just lists the top-level dependencies I know about.

* There's a `requirements.txt` file generated via pip-tools that lists all the dependencies with explicit version numbers.

* The bash script then makes sure there's a virtual environment in that folder & the installed package list matches exactly the `requirements.txt` file (i.e. any extra packages are uninstalled, any missing/mismatched version packages are installed correctly).

This was great because during development if you want to add a new dependency or change the installed version (i.e. pip-compile -U to update the dependency set), it didn't matter what the build server had & could test any diff independently & inexpensively. When developers pulled a new revision, they didn't have to muck about with the virtualenv - they could just launch the script without thinking about python dependencies. Finally, unrelated pieces of code would have their own dependency chains so there wasn't even a global project-wide set of dependencies (e.g. if 1 tool depends on component A, the other tools don't need to).

I viewed the lack of `setup.py` as a good thing - deploying new versions of tools was a git push away rather than relying on chef or having users install new versions manually.

This was the smoothest setup I've ever used for running python from source without adopting something like Bazel/BUCK (which add a lot of complexity for ingesting new dependencies as you can't leverage pip & they don't support running the python scripts in-place).

It does a bad job of dealing with versioning conflicts and multiple projects, so Python developers resort to hacks like virtual environments to get work done. Compared to Cargo or even Go modules, it's not a great solution. It's also missing lots of features that are standard in other package managers.

> so Python developers resort to hacks like virtual environments to get work done

It's really hard with many deps, it's why cabal (for instance) moved away from a global model.

The need for a virtualenv has nothing to do with pip. Python only has “global” dependencies due to the way its import system works.

Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics).

Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).

so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.

Sort of. The equivalent of "virtualenv" would be more "rvm gemsets", which is what people did before bundler. Bundler is doing something different.

Bundler is not a "node_modules" style setup. It does not require dependencies to be in a local path (although they can be, the default is they live in a system-wide location, and this does not limit functionality). It also does not support more than one version of a dependency in the same execution environment (as node_modules does) -- that really would be impossible in ruby too.

It's possible something about python's design would make the bundler approach impossible, I don't know. But it's not "dependencies are installed globally" alone, as that's true of ruby too.

We would probably all benefit in understanding better how these things are handled in other environments. And I include myself here. I think ruby's bundler really set a new standard for best practices here, and many subsequent managers (like cargo) were heavily influenced by it, although many don't realize it. But meanwhile many don't even realize what they are missing or what's possible.

Like the basic idea of having a specification of top-level dependencies (including allowable ranges) separate from a "lockfile" of exact versions in use of ALL dependencies... is just so hugely useful I never want to do without it, and I think is compatible with just about any architecture, and yet somehow JS is still only slowly catching on to it.

> Ruby works the same way

Kind of, if you ignore Rubygems, which is also part of stdlib at a lower level then bundler (and also, originally wasn't.)

> but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies.

It can do that because rubygems manages multiple installed versions of packages and allows per-project ("per call to require", potentially, IIRC) specification of which one to pull from the globally-installed versions (this was originally done by monkey patching require when rubygems was an add-on.) This lets bundler easily live on top of it providing per-project dependencies somewhat more smoothly than Rubygems does without requiring anything like a venv.

> Perhaps ruby was more "hackable" by bundler.

Ruby is ludicrously hackable, yes.

> (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).

Rubygems also wasn't part of stdlib originally, and started out relying on hacking around the way Kernel#require works.

> It can do that because rubygems manages multiple installed versions of packages

Oh wow, the default python dependency manager only lets you have one version of each package installed system-wide?

Yeah, that is a limitation. As opposed to rubygems (the first dependency manager although as you say not originally built-in to ruby) which has system-wide install, but always let you have more than one version installed.

Without fixing that one way or another, there's no sensible way, true. virtualenv is certainly one way to fix it. I wonder if there would have been a more rubygems way to fix it.

Honestly I found the multiple versions approach more complex, confusing and more hacky. A virtualenv is just “node_modules” that also contains a Python executable.

It’s a directory - you delete and create them at will, fast, and don’t worry or care about the system Python. Having some crazy setup that patches “require” to handle concurrently installed package versions seems insane, especially if you cannot actually use them concurrently in the same Ruby process. So, segmenting them by project (aka virtualenv) seems like the best solution.

Not quite true. Python import mechanics are quite hackable and controllable programmatically and externally via controlling the relevant PATH env variables. It's just that no one bothers with it and instead seems to rely on the set of global folder-path lookup mechanics that are standard.

But nothing stops you from masking global-libraries with local library versions (similar to node_modules). Why hasn't anyone done this you may ask? I don't know the answer to that.

There are a few reasons you can’t have multiple versions of the same module at the same time. Consider a simple enum class defined in a package: two versions now have two different enum objects which may not compare equally. Maybe a function from module A would return Enum.A and pass it to module B, which would then compare A.Enum.A to B.Enum.A, which fails. Super confusing.

So yes, Python’s import system is dynamic enough to do crazy things, but I don’t see how we can ever retrofit that into Python.

Regarding dependencies installed into a project-local directory (node_modules): that’s a virtual environment. Just more flexible.

It may depend on you experience with each language. I have much more experience with Python than Go (almost none with Rust) and therefore I have much better time with python packaging tools (I don't remember a single issue, that I didn't find a satisfactory solution -- as much as possible in the packaging world with myriad conflicting use-cases)

For example, my experience with `cargo` (that I mostly use to install command-line utilities such as rg, fd, dust): it is great when it works as written in the instructions but sometimes it doesn't (running `cargo` may involve a lot of compiling -- in contrast to `pip` which can use wheels transparently and avoid compiling even for modules with C extensions -- I guess there might be a way to do something similar with `cargo` though not by default).

What's wrong with virtual environments? The required tools come bundled with Python nowadays and are super easy to use.

Agreed 100%,in fact of the languages I work with regularly (Python, Java, C#, JavaScript, Go) Python has the simplest dependency management solution via virtualenv + pypi + pip. Not sure why every Python thread turns into a conversation about the pain of Python dep management... seems overblown.

Hm.... Let me see. The number of operations I have to perform in Maven vs Python for independent packages:

Maven Scala project - create skeleton, add libraries to POM, write app, run app

PIP Venv Python project - create venv, enable venv, create requirements file, write app, run pip to install dependencies(possibly install GCC and extra libraries), run app

(Oh... and god forbid that you forget to deactivate venv)

You're lying when you say that library management is easier in Python. It's just factually untrue.

Most of the steps you're listing take only a few seconds. They're talking about the actual management, not how long it takes you to type . venv/bin/activate

A few seconds here and a few seconds there - it's death by a thousand papercuts.

There's no community consensus - that keeps Python from advancing to where it needs to be.

I said it once and I'll say it again - Python lacks mature tooling.

You can't have death by a thousand papercuts when there are exactly three delay-papercuts adjacent to a step that takes a significant amount of time.

There are contexts where little delays matter, and you didn't pick one of those.

You don't have to activate the environment. I never do; it's a strictly optional convenience (it you think it's convenient).

Instead, simply run the interpreter installed in the environment when you run your app, e.g. "./my_env/bin/python my_app.py", and things will just work. No activation required, no special mode, nothing to forget.

The part about requirements.txt and installing packages could also be simplified if you did it the other way around: install first and create the requirements file from that:

  $ python3 -m venv my_env
  $ my_env/bin/pip install some-dependency
  $ my_env/bin/pip freeze >requirements.txt
  $ my_env/bin/python3 my_app.py

There you go. Setup, install and run in four steps and zero modes.

That's literally one step more, than Maven.

That's before you get to package your app...

Maven is far more complicated and burdensome when compared to virtualenv and pip... pom.xml, what a righteous mess of XML and overly specified nonsense.

This is just to run your app, to package it - it's a whole different headache in Python.

It's literally a `mvn jar` and that's it!

The biggest piece that is missing is you have to go out of your way to get sane dependency management. I have only used JS and F# (same dependency management as C#) out of your mentions, and it's the official tools that enable the local dependency management with a single command ("npm install X" or "dotnet add package X").

If you're using python, you don't know to check out pyenv/etc until you have a huge mess on your computer due to pip's behavior.

GP comment was objecting to "stupid" package managers. Maybe they think pip is stupid, because it's unquestionably a package manager.

It's a package manager in the vein of "old-school" package managers that came from Linux distros and whatnot. It maintains a global dependency chain across your entire machine. This can be good for security fixes in that you only have 1 copy of a package & everyone references it. This is not good for development because it doesn't provide a sandbox'ed environment for you to do development in (ala cargo as others have mentioned). It also causes issue if you try to install 2 packages but they rely on incompatible versions of a popular package meaning you have to choose which package you want installed.

Some of this has been mitigated with virtualenv but having a project express it's packages & have that automatically reflected in the environment.

Finally, Cargo to my knowledge actually lets multiple dependencies exist (even within the same project!!!) so that you can have a dependency like:

                        dep1 -- dep3 <= v1.6
                      /

< my awesome project> -------- dep3 >= 3.0 \ dep2 -- dep3 >= 2.0

That's not possible if you don't have the right language hooks because module resolution needs to be aware of the version of the library (i.e. when you go `import numpy`, it actually needs to be aware of the package it's being imported from to resolve that correctly).

Now whether or not it's a good idea to support this kind of dependency stuff can be controversial. In practice though clearly it does cause problems the larger your codebase gets as you're more likely to have some nested dependency chain that's time-consuming to upgrade so you'd rather move faster than make sure you're only running 1 version of the dependency.

For some reason I've run into this type of dependency chain issue many times in JS, but have never run into it in Python despite using both languages pretty heavily. Maybe because the JS ethos is to change things so quickly, if you're not making a major breaking change to your package's API every year or two then you're getting left behind (only kind of joking). Also probably because the standard library is so small in JS (or at least it used to be, and may projects want to be compatible with at least some older browsers) so the average number of dependencies that a typical library has is probably much higher than in Python.

I really don't get the fuss about the global dependency management though, maybe I would change my mind if Python shipedp a great implementation of it. But I feel like the problem is already solvable in multiple ways with containers, VMs, or virtualenvs and I don't think yet another abstraction to separate environments would add much value to my day to day workflow building python apps.

This is my experience too. I’ve never actually encountered a global dependency conflict with Python pip though in theory it’s possible. But I have encountered the same version conflict problem in dependency chains that exists with Node npm.

And yet, I hear so many more complaints about Python pip and I really don’t understand the disconnect. Perhaps dislike of pip is actually triggered by usability issues? And then people look for other reasons to explain their dislike?

It happens with the data science/scientific programming stack a lot, at least twice in the last month I've pulled in a small dependency that changed my numpy version which broke everything.

Thanks. I suppose it could be related to the rate of change in the ecosystem. Python’s data science / scientific programming stack definitely changes faster than Python’s web stack which is currently my primary use case.

As mentioned elsewhere in this thread, resolving this dependency issue would require a change in the Python language itself.

Maybe you just don't rely on packages that have different update cycles.

If you're only working with the standard library, boto3, twisted and redis - you're unlikely to have issues. You get into big issues, when you get to more obscure libraries... or libraries that are C bindings.

> It maintains a global dependency chain across your entire machine.

There are per python binary, per user, per virtualenv installation (per project or per whatever you like) that make the conflict less likely.

Sometime packages "vendor" their dependencies e.g., there is `pip._vendor.requests` (thus you may have different `requests` versions in the same environment).

There were setuptools' multi-version installs https://packaging.python.org/guides/multi-version-installs/ (I don't remember using it explicitly ever -- no need)

> Finally, Cargo to my knowledge actually lets multiple dependencies exist

That's a bug, not a feature. It enables sloppy development and the disasters like on NPM

How so? The version ends up being part of the type, so a type from two different versions of a given package are not compatible, which solves most if not all of the issues.

If a package is just using a particular library internally, I don’t see why the package manager should prevent using it with another library that depends on a different version.

> I don’t see why the package manager should prevent using it with another library that depends on a different version.

I do. The main reason for Linux distributions to exists is to provide a development and running environment where:

- API/ABIs do not change for the whole lifetime of the distribution. No new features, no new bugs, no new vulnerabilities, so that your production code can run reliably for 5+ years.

- Vulnerabilities are fixed with minimally invasive patches.

- Vulnerabilities are fixed in reasonable times even if the upstream development stopped. Patches are well tested against the set of packages in the distribution.

You simply cannot have these 3 features together if a distribution ships 10 different version of each library.

It's already a ton of work to maintain packages in stable distributions.

I’m confused by your comment. This is about a programming language package manager, not an OS package manager. Or was that just an example?

As I said above:

> Now whether or not it's a good idea to support this kind of dependency stuff can be controversial. In practice though clearly it does cause problems the larger your codebase gets as you're more likely to have some nested dependency chain that's time-consuming to upgrade so you'd rather move faster than make sure you're only running 1 version of the dependency.

Consider the view that some times it can be sloppy & other times it's not & it's impossible to distinguish between the two in an automated fashion.

> Finally, Cargo to my knowledge actually lets multiple dependencies exist (even within the same project!!!)

That's not pip's fault, that's Python's fault. Python's module system has no concept of versioning, so there can only ever be one copy of a module that has a given name.

And this is an interpreter detail that is exposed through the language itself, so it can't be fixed without causing severe pain.

Honestly - My non-polite, personal impression of all the complaints is that they're borne from a very specific development environment. One that includes lots of dependencies similar to "npm", wants "docker style" local development, and the devs seem to think dependency-management is hard and you need complicated "semantic" versioning and version operators... but really it's just because they're working in a complex ecosystem of microservices.

But, for the other 99% projects though: Most of their dependencies won't break compatibility, you'll never uncover a hard version dependency that the package manager can't solve, you'll never need to "freeze" your dependency versions and you can pretty much just rely on a semi-persistent environment with all your necessary packages installed and semi-regularly updated. Essentially smooth-sailing.

Microsoft already bought NPM so there's precedent.

Yeah, but do we REALLY want M$ to control, github, npm, node, rust, python, and ...

We still have Idris and ATS.

Considering that half of all prominent functional PL researchers are hired by Microsoft Research I won't be too confident about that.

M"$"? 1999 called, they want their edginess back...

> M"$"? 1999 called, they want their edginess back...

You realize that we're potentially just one leadership change away from returning to the 'bad old days', right?

You're missing the big picture. In 1999, Microsoft's revenue was at least 10x that of the closest competitor.

It controlled the client (desktop) and was working on controlling the server, too.

Google, Facebook, Netflix, etc didn't exist. Amazon was much smaller and in a different niche. Apple was trying to not keel over in 2 months.

We're not a leadership change away from anything. The world has changed.

> We're not a leadership change away from anything. The world has changed.

The world has changed, but I didn't mean "the bad old days when Microsoft had a stranglehold over the industry" but just "the bad old days of an evil Microsoft". Their market power isn't relevant to whether they are a bad actor, only to how large their impact is as a bad actor.

pip is completely fine

pip in virtualenv. It is solved problem.

Not even pip developers believe "pip in virtualenv" makes python dependencies and package management a "solved problem"...

90% of professional python developers believe that python dependencies is a solved problem (with pip and virtualenv).

The rest is too vocal.

>90% of professional python developers believe that python dependencies is a solved problem (with pip and virtualenv).

99% of professional python developers think that you've pulled this statistic out of your ass!

> >90% of professional python developers believe that python dependencies is a solved problem (with pip and virtualenv).

> 99% of professional python developers think that you've pulled this statistic out of your ass!

72.6% of all statistics are made up, anyway.

> 90% of professional python developers believe that python dependencies is a solved problem (with pip and virtualenv).

I doubt that even 90% of those professional python developers who believe that python dependencies is a solved problem believe that it is solved with pip and virtualenv; the conda faction has to be bigger than 10%. Plus there's the people that think pip/venv aren't enough, but that tools on top of them plug the gaps (poetry).

But I think that the share of professional developers who see it as a solved problem at all is less than 90%. Obviously, we've all got some way of working with/around the issues, that doesn't mean that we don't feel that they exist.

If I can give my anecdote, the last 3 companies I worked at that were heavily using Python and the hundreds of developers in them were all relying on pip and virtualenv. And it worked just fine no matter what the HN crowd would have you believe.

conda had minor usage in the last one for building a handful of special projects mixing C++ and Python code (highly specific code in the finance industry), after build the artifacts could go into the python repository (internal pypi) and be usable with pip. Everything was down to pip at the end of the day. As a matter of fact, the guys who used and pushed for conda were also the ones pushing the hardest for pip because pip is the answer to everything.

Well, if the consumers are all devs, then sure.

Well - you can add one more to your anecdote - my current job is 13 active developers, 200+ megabytes of 1500+ .py files developed over 9 years by 30+ developers. It's all virtualenv/wrappers + pip.

Our data scientists like Conda - but our developers don't touch it.

My pet conspiracy theory is that Docker was created in part because Python code is un-shipable otherwise.

Spot on!

I'm talking about Susie Q and Joe Sixer. The amount of fiddling with package systems, build, and container systems is an anti-pattern. One guy on a small team sets up and controls the stuffs. Individual contributors shouldn't be messing with the env or putting new packages in :)

I often wonder what problem is being solved by Poetry, that pip with virtualenv(+wrappers) doesn't solve perfectly well. requirements.txt ensures you have the right version of everything in production, and virtualenv lets you explore various versions of libraries without risking any destabilization of your development environment.

I struggle to see what spending time looking at Poetry will yield in terms of any actual benefit, though I would love to be informed/educated otherwise.

virtualenv always feels like a hack to me. Too many times I’ve forgotten to activate the virtualenv halfway through a project and now I’m troubleshooting all the problems I just caused with some packages installed in the virtualenv and some globally and oh half of them aren’t in my packages.txt so now I can’t remember which package I needed for this...

I don’t expect my dev tools to be idiot proof, but they should at least try to be “I’ve been hacking for 18 hours straight and I just need to commit this last line and I can finally go to bed” proof.

One thing that helps is: # Don't let people touch anything in the root environment export PIP_REQUIRE_VIRTUALENV=true

That prevents you from ever doing a pip install in your root environment.

I've got about 20 different pip environments, and virtualenvwrappers (workon xxxx) makes it pretty seamless for me to hop back and forth between sessions. I'm also pretty dedicated to doing all my work in tmux windows - so my state in a half dozen virtualenvs is changed by changing my window (which I've done a workon)

I guess what I'm really interested in is, "The last three years of my life I've used virtualenv/wrappers + pip, and haven't run into any problems. What can Poetry do for me, and why should I change my work habits? Genuinely interested in using new and better tools.

Same here. 20 years on python and I've never done anything more than clone a repo, venv and pip install -r requirements. And then cd && python -m to run the source code, maybe set PYTHONPATH in some cases.

I don't think I've even written a setup.py .

Obviously there's a whole world of development and deployment where these things are relevant, but there's also a massive world where nobody even understands what they are missing.

My current gig has a 9 year python repo, 3526 .py files over 228 megabytes, 13 (current) developers. It's all managed with virtuanenv/pip install. So - even larger projects seem to get by okay - would love to read something on Poetry that just says, "Here is why it's a lot better than virtualenv/pip"

Agreed. I don't get the fuzz about pip/setuptools/virtualenv. I have shipped and deployed Python code countless times, never encountered an issue a `setup.py` couldn't solve.

It is, unless you want to ship.

Just ship a container to the server.

Hmmm... Let me check if all of my users run a server on their computers.

Sure... Ship that docker to Spark master node and... do what exactly?

I'd love to see Poetry take off. I'm watching it pretty closely.

Same here. Poetry has been a joy, after many bouts of frustration with pipenv.

Same, we switched from pipenv ~6 months ago, had not had to worry about package/env related stuff since then, "just works".

"evangelized" a sibling team also to consider switching, they were sceptical but just recently they mentioned they also like it more.

Better not move to NuGeT.eXe

We use poetry for apps in production. At this point I think that's the winning solution and as it continues to grow and improve I think it will overtake all the others in this respect.

Yes, Poetry is great! I avoided Python for a long time due to it's bad package management/environment handling situation, but Poetry solved all my problems there.

People keep saying that about every new solution. But then another one comes along that's even better-er, and the previous one peters out.

The biggest need for a package manager and its ecosystem is continuity: the stance that new features and paradigms will be gradually shifted toward — without package-ecosystem incompatibilities, without CLI commands just disappearing (but instead, with long deprecation timelines), etc.

In other words, an officially-blessed package manager is one where, when something better-er comes along, it gets absorbed by the existing thing, instead of replacing it.

That is what the Python ecosystem is missing.

Yes, Poetry should be the blessed package manager.

I only want Poetry to become the be-all-end-all of package managers if it turns out that Python really is never going to fix the core problems that have engendered so many of the hacks upon which Poetry (and its competitors) is precariously balanced. Pyenv and venv, for example.

If you were doing a green field redesign, how would you want Python to fix the core problems?

> Yes, Poetry should be the blessed package manager.

Last time I tried to use poetry (and this is why it was the last time I tried to use poetry), it ignored global pip settings and had no documented mechanism for its own settings (I believe poetry uses its own implementation of or captive install of pip) which made it completely unusable in a corporate environment with annoying SSL interception issues to work around where pip + venv worked.

Poetry is a much smoother experience when it works, though.

It will install a virtual env if you don't have one active and it will use the active one if you do. What global pip settings, for example?

Although I generally like it, the two major issues I have with poetry are the abysmal dependency resolution times and the handling of binary wheels.

I don’t think it’s that another one comes along that’s better as much as it is the new “better” ends up missing some important corner case. Pipenv advertised itself as solving all problems, but once someone tries it in practice, they realize that it introduces a new problem: every little interaction takes literally 30 minutes for any non-toy project. I’ve heard mixed things about poetry, but I wouldn’t be surprised in the least of it failed to behave as advertised just because this has been my experience with every package manager I’ve tried to use. And it’s embarrassing when every other language has a package manager that just works.

EDIT: It was probably misleading to characterize Pipenv as advertising itself as solving all problems; it’s probably more correct to say that it’s significant weaknesses weren’t advertised and thus one has to invest considerably before discovering them for oneself.

Just a heads up to anyone who hasn’t looked recently: pipenv has been very actively worked on since earlier this year and has had four updates that fix a lot of issues. Earlier this year I would have said Poetry is better hands down, but after the updates and after using poetry and seeing some of its quirks, it’s a much closer matchup.

If it wasn't so opinionated it might have been more successful.

Just one example: you want your virtualenvs to be created in ~/.virtualenvs so that pipenv is a drop-in replacement for virtualenvwrapper+pip? Tough luck for you, Kenneth Reitz doesn't think that's how it should be done.

At least 3 or 4 times some issue I've wanted resolved I found in the issue tracker with the last message "we'll have to check with kennethreitz42 whether we're allowed to change that" and then silence for a year.

It could still catch up with poetry, but from what I've seen there's a fundamental mindset difference in how change requests are approached between pipenv and poetry.

Last I checked (3-4 month ago) Pipenv only cares about the situation when you are deploying code on machines (or containers) you have complete control over. If you're writing code for deploying on machines you don't have control over, via for example pip install, then pipenv isn't helpful while poetry supports this out of the box.

Interesting. I haven't used pipenv on any very large projects, but I'm surprised to hear about the slowness. With the (admittedly small) projects I've tried it, I found that it does more or less just work.

As I understand it, the problem is that Pipenv needs to resolve the dependency tree to do just about anything; however, the dependency tree is dynamic—to determine a package’s dependencies, you have to download the package and run it’s setup.py. To get the whole tree, you have to recursively download and run each package. So the cost is proportional to the size of the dependency tree, so it’s very plausible that it works fine for the smallest projects.

> People keep saying that about every new solution. But then another one comes along that's even better-er, and the previous one peters out.

I think this is happening these days frequently. People try to cover all use cases and then end up in biting more than they can chew. It won't work that way. Good set of MINIMALS, is easy to maintain, sustain and extend.

What the Python ecosystem actually needs is a single opinionated perspective on versioning that is followed by everyone, such as NPM's semantic versioning. In the absence of that I don't see how dependency resolution and thus packaging is ever going to improve in Python.

Guido is very opinionated...

He just doesn't care about package management.

> The biggest need for a package manager and its ecosystem is continuity: the stance that new features and paradigms will be gradually shifted toward — without package-ecosystem incompatibilities, without CLI commands just disappearing (but instead, with long deprecation timelines), etc.

I disagree. I used to think that that's the problem, but having seen a few more cycles of it, the problem isn't that kind of commitment - after all, the whole python ecosystem enthusiastically jumps into the new thing, and Python people are used to relatively short deprecation cycles. The problems are the actual problems; every Python package manager is just embarrassingly awfully bad as soon as you try to use it for 5 minutes, presumably because they're developed by Python people who've never used a decent package manager and so think that no-one could ever need deterministic dependency resolution, once you've pinned a transitive dependency there surely wouldn't be any reason to ever want to unpin it, having the package manager coupled to the language version is absolutely fine, no-one could ever want a standard way to run tests ...

Much of Python's growth has been driven by data science. Here, the conda package manager is pretty ubiquitous. Conda packages system and other non-Python dependencies (such as the CUDA SDK), removing the need for data scientists to resolve these non-trivial dependencies themselves. This is likely unneeded/ unwanted for production web app deployments.

Given the varied use cases for Python, the goal of a single package manager may be misguided.

My understanding is that the people who developed Conda would love to have stuck with pip, and originally wanted to see about upgrading pip to support their use cases. And it was GvR himself who told them that that wasn't going to happen.

That was a long time ago, though, when scientific computing was a small niche for Python. It might have been reasonable to say it's not worthwhile to take on all that extra work just to support the needs of a small minority of users. Fast forward the better part of a decade, and it turns out that scientific computing did not stay a small niche. I think that one could make a strong argument that, in retrospect, that brush-off did not end up ultimately serving the best interests of the Python community. It made the community more fragmentary, in a way that divided, and therefore hindered, efforts at addressing what has proven to be one of Python's biggest pain points.

Conda predates pip by perhaps a decade.

Really? The first release of pip was in 2011 [1] and the earliest release of Conda I can find is 1.1.0 in Nov. 2012 [2], and the first public commit (into an empty repo) was a month earlier [3].

[1] https://en.wikipedia.org/wiki/Pip_(package_manager)

[2] https://github.com/conda/conda/tags?after=1.3.0

[3] https://github.com/conda/conda/commit/c9aea053d8619e1754b24b...

May be anaconda that I'm thinking of.

Anaconda was released in 2012, as well. Conda is a tool that is part of anaconda.

This one: https://en.wikipedia.org/wiki/Anaconda_(installer)

I forgive myself, it's pretty confusing :D.

Ah. Name collisions suck.

Especially when they conceptually do the same thing.

And that's another issue with Python ecosystem

Why do you think Python is more susceptible than other platforms?

It is true that PyPi was designed before the author/project naming scheme popularized by github. Other than that I don't see a greater problem with name collisions in Python.

Susceptible - yes, all platforms could fall to this.

That's why a strong leadership in a community, or subcommunities, works well. Python lacked this leadership, that leads to millions of half-arsed projects that compete... without moving the whole platform forward. It feels like NIH syndrome has permeated Python. Hopefully that's going to change

I also use poetry for everything. I have 0 problems, things work on my mac, my interns pc, aws instances, I don't even see what problem people are having. Before that I was using pipenv, and before that just good old requirements.txt - there were a few occasional issues, but really not much even then. At this point, I suspect it is more about regurgitating a complaint than a real issue. But, I could be lucky and completely wrong...

If you don't need c or c++ dependencies it's ok. If you do, it's very very painful. To be fair, most of the DS libraries can be handled by conda, but if you need both conda and pip, then you're going to have a bad time. (Source: this is my life right now).

Oh man, this is my life right now, too. In my case, we're using tensorflow or tensorflow-gpu, depending on the host system and, unfortunately, only Conda offers tensorflow-gpu with built-in CUDA. Add to this that the tensorflow packages themselves are notoriously bad at specifying dependencies and that different versions of tensorflow(-gpu) are available on conda-forge, depending on your OS.

Tensorflow is the worst (along with ReAgent from FB).

I think it's because they have their own internal build systems, but they never play well with pip/conda et al.

One of my recent breakages was installing the recsim package, which pulled in tensorflow and broke my entire app. There's actually a recsim-no-tf package on PyPi, presumably because this happens to loads of people.

I see, I miss a lot of issues as don't use any GPU stuff, mainly flask + scipy and friends.... probably this is what saves me.

It's not even the GPU versions, even the CPU stuff causes issues.

The core problem is that pip will happily overwrite your existing dependencies when you attempt to install a new package.

- until a few months ago no way to sync an environment with a lockfile (remove packages that shouldn't be there)

- no way to check if the lock file is up to date with the toml file

- no way to install packages from source if the version number is calculated (this will likely never be fixed as it's a design decision to use static package metadata insetad of setup.py, but is an incompatibility with pip)

- no way to have handle multiple environments: you get dependencies and dev-dependencies and that's it. You can fake it with extras, but it's a hack

- if you upgrade to a new python minor version you also have to upgrade to the latest poetry version or things just fail (Something to do with the correct selection of vendored dependencies. May have since been fixed -- new python versions don't come out all that often for me to run into it. And in fairness the latest pip is typically bundled with each python so it avoids that issue)

I still use poetry because it's more standard than hand-rolled pip freeze wrapper scripts, and there's definitely progress (the inability to sync packages was a hard requirement for me but is not fixed) but it's not quite there yet

Interesting, i usually rebuild my env from pacakges so don't notice 1,2, or 3. I guess 2 should be fixable by poetry by including more from the toml in the lock file. Point 4 also didn't bother me as I in general just have the main and dev deps, this seems an easier thing to fix for poetry though. I actually have encountered 5 when fiddling around with pyenv.

If you don't know what the problems with pipenv or requirements.txt were, you're really not qualified to judge whether poetry has solved them or not.

You are reading wrong, it did solve my issues with reqs and pipenv. Also you certainly aren’t qualified to judge my qualifications.

I wouldn't advise using anything other than the tools blessed by the PSF for mission-critical stuff. Using Poetry for local development is fine but don't build a huge infrastructure around it and don't use in production.

I migrated the CI/CD of my company to Poetry some time ago, it worked fine for some time until we needed a feature that Poetry didn't support. I submitted a PR adding the feature to Poetry but their sole developer was apparently taking some time off and the project remained without any development for several months.

I migrated the CI/CD to use my own Poetry fork but it was very cumbersome, Poetry has a very weird build system so forking it is not simple.

At this point, I realized that I was just wasting time. There is nothing that Poetry does that the other (old and stable) tools don't do. Poetry was the result of me falling for the shiny toy syndrome.

Conda works well.

Have never used cargo - what can cargo do that conda cannot?

Conda works slightly better than pip, which is a pretty low bar. Python package management is probably the absolute worst thing about that language.

Still, it worked perfectly for me for Python, and a few more things. So I ask - what problems does conda have? And which of those does cargo not have?

Alternatively, what does cargo do better than conda if they are not feature-for-feature comparable ?

conda can manage python versions itself as if it was another package.

https://docs.conda.io/projects/conda/en/latest/user-guide/ta...

i've never used it particular feature, but then i'm using python since 1.5 and i'm just used to it being a bit behind the times. stockholm syndrome, you might say, especially after trying out rust and seeing the work of art cargo is.

How do you ensure a team of 30 developers are all using the same version of python and all of the dependencies of your project with conda? How do you distinguish prod dependencies from dev dependencies? How do you update and manage transitive dependencies?

There are more, but those are the big three.

Conda manages Python itself as one of the dependencies - so that’s not actually a problem.

I used conda to manage 2.6, 2.7 and 3.3 side by side, a d that was fine. I never locked the patch version (e.g. 2.7.3 vs 2.7.5) though that is definitely possible.

It apparently requires a specific workflow - explicitly editing the requirements.txt file, rather than freezing it - which is not harder and which I was doing since day 1 but is apparently uncommon.

(And it worked well across a few tens of machines, some running windows and some running Linux, with a mix of os versions and distributions. So I know it worked well already in 2013, and I’m sure it works better now).

Speed is not good, but I never had it take more than a minute for anything. Some people here are reporting 15 minutes resolution times - that is a real problem.

You didn’t answer the issues I raised.

Actually I did.

You make sure everyone uses the same Python version by setting it as a dependency. I mentioned I only set the dependencies on minor version (e.g. 2.7) rather than path version (e.g. 2.7.3), but the latter is supported - for the python version as well as for any other package.

You make sure the exact versions you want are in use by editing and curating the requirements.txt rather than freezing it. It really is that simple, but somehow that's not a common workflow.

prod vs. dev is the only one I didn't address because I don't have experience with that - but I know people who manage "requirements_prod.txt" vs "requirements_dev.txt" and it seems to work for them.

How do you manage the specific version of transitive dependencies? For example, I install library A and it requires library B, but your code doesn’t use B directly?

A better question, what’s your workflow for installing a dependency and 6 months later updating that dependency?

Just put them in your requirements.txt even if you don’t depend on them directly.

That’s basically the whole idea of conda’s solver, I think. If your list works - fine. If n9t, it will find a set of package installs that makes it work.

I guess that’s also why my resolution times are way faster than what some people describe.

I treat requirements.txt closer to a manually curated lock file than a wishlist (which most req.txt files in the wild are).

This is a giant pain in the ass compared to every other modern language. Either you’re manually managing precise versions for all direct and transitive dependencies or you aren’t running exactly the same versions of libraries across team members and environments which would be horrifying.

There is a command (not freeze) that dumps the exact versions deployed, similar to a lock file. I never used it, because it doesn’t work well across operating systems (e.g. do it on Windows and try to use it on Linux). This is less of a Conda problem and more of a Python problem - same package name/version has different dependencies on different architectures.

But manually curating the dependencies has been painless and works fine for me and my team for almost a decade now.

I prefer that my tools do the busy time consuming manual work when possible. Every other language I use has better tooling than python for this.

And they were the biggest headaches on my team, bar none, of them all. An absolute nightmare to coordinate--and it seemed like even minor things that should be unrelated, e.g., a locally installed system (not python) package slightly off-version, could eat a developer's time trying to resolve.

This is probably a hard problem, but the dependency graph resolution in conda is a thing of nightmares. It is so appallingly slow even for relatively menial tasks that updating environments becomes an exercise in frustration.

I'm unsure what are the core issues, but in my experience cargo was always pretty quick to use and if it fails, it fails fast.

Conda on the other hand is slow for the simple cases, and if the graph becomes complex it will just churn for 15 minutes before throwing it's hands in the air and giving up with some cryptic error.

I suspect it comes back to the fact that packaging and dependency management was thought about upfront for Rust and the whole ecosystem was built well from the get go?

I've been meaning to write a blog post on this topic. That is, on the various speeds of various package managers. I don't know a ton about Conda, but when I was looking into Poetry, one of the core issues is: https://python-poetry.org/docs/faq/#why-is-the-dependency-re...

> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.

Cargo doesn't need to do anything with packages directly to do its job; everything it needs is in the index. This makes it pretty fast.

>> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.

This sounds like something that could be done server side, either by PyPI or another entity and expose it through a new API endpoint, instead of doing it on every Python developer's machine.

The core issue is that setup.py can be non-deterministic, so them doing it server side may not give the right results.

https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-k...

conda taking 15+ mins to resolve dependencies is by far its biggest weakness.

If it doesn't fail to "solve the environment" after 30 minutes that's already a win.

In addition to the other replies, one I've encountered is the case where conda doesn't provide a build for a package (thus, one must use pip or something else to manage that dependency) causing weird issues with respect to maintenance and updates.

The two worst are in the other comments: ensuring sync'd dependencies across multiple environments and developers, and the horrendous resolution times leading to a useless error message when failures occur.

Just work, instead of not working? https://github.com/conda/conda/issues/9059

Wow, color me impressed /s.

You found a bug report for a problem specifically on fish on MacOS, which can be resolved by "deactivate / reactivate" according to the discussion.

Were you trying to say something?

>> which can be resolved by "deactivate / reactivate"

Nope.

Feature selection, dependency overriding, worspaces, to name a few.

conda has "environments" which are isolated from each other and can each have specific versions of python and dependencies installed in them (and each has their own "pip" local installation as well, in case something wasn't specifically packaged for conda).

What are those "workspaces" you refer to?

What is "feature selection"?

If only there was a way to find out about such things... (wistfully looks into the distance)

> An official package manager with great dependency resolution would be fantastic.

Need something like cabal. And a package index.

Python’s biggest hurdle is and always will be speed. Pip and wheels aren’t great but I would rather stop using golang. That’s a bigger win for me to stop maintaining proficiency and tooling for a language that I honestly consider inferior for the way I think about problems.

Strongly disagree. Python’s adoption is primarily driven by how easy it is to get started (barring some nasties in datetime, typing, and ofcourse the package manager).

Python is plenty fast for most automation tasks.

Python is fast enough for many tasks, including number crunching for which it excels (as long as you are using the right libraries).

Projects like Cython allow to tweak without too much effort the parts of a program that need an extra-boost.

Last but not least, there have been discussions in the Python community in the last weeks of ways to speed up considerably the default (CPython) implementation.

Hopefully, all of this will bear some fruits in the next 2-5 years. Guido will probably help from his new position at Microsoft.

> discussions in the Python community in the last weeks of ways to speed up CPython

This piqued my interest. I found this, by Mark Shannon: https://mail.python.org/archives/list/python-dev@python.org/...

Is that what you mean?

Only because libraries have to use C bindings to make it fast enough. I don't think Python performance is good enough when I have to stop writing in the language that we are benchmarking to get good results.

You are talking past the point I am making. No doubt Python has excellent adoption because the language is incredibly idiomatic and easy to understand -- thus why it is my favorite language. Python is great in all of the ways you just stated but it could be better and I am saying that speed for CPU bound tasks is the way it could be better. I am so tired of binning Python in favor of golang the moment latency becomes important, I would like to use it all the time.

Python's biggest hurdle is it's lack of good tooling.

Try building a Unix app, with data in standard Unix locations(bin, shared, lib, etc) and you'll find that you have to write custom code.

And Google searches don't help :(

I like the way npm does. It has a cache of all downloaded versions and it tries to flatten the dependencies when possible. If there are incompatible versions they remain deep in the dependency hierarchy.

Npm has so many woes though. Love the idea but the execution leaves much to be desired.

Like what?

Can you go deeper on this, honestly interested. I use NPM all the time have not run into too many issues.

I jumped down the black hole of pip + geo packages this week. Your comments are absolutely spot onm

Package management and performance. Microsoft has done an impressive job of making software development better lately, and if any community needs it, it’s Python. Python has a lot of potential, but it’s sorely hampered by poor package management and performance, which are ultimately both symptoms of its commitment to exposing the entire CPython interpreter as a stable public interface. If the Python leadership were willing to commit to a narrower interface that met the goals of the c-extension community and also allowed Python to improve with respect to performance and package management (and I think this is very possible, hpy gets us close afaict), then we could have the best of both worlds, and I’m optimistic that Microsoft could provide this leadership.

What poor performance?

At this point, this criticism has become a "thing" that no one really expands on or enumerates as if it's a given. It's not a given, and at the very least is nuanced and complicated.

I’m not sure where you’re getting your information, but Python itself is quite slow (100-1000X slower than Go or Java) and it’s only “fast” when it’s dispatching to C or Rust or FORTRAN, and even then (to your point about nuance) these optimizations are only feasible sometimes; namely, when the cost of serializing is lower than the efficiencies gained by the optimizations available in the target language. This is all pretty widely discussed, including here on this very forum.

No doubt that Python is sometimes fast enough (e.g., vanilla CRUD apps that do all of the heavy lifting in Postgres), but sometimes it’s not and you’re left with really crumby optimization options. And since we rarely can know with certainty on the outset of a project whether or not the bottlenecks will be amenable to the optimizations afforded by Python, it’s a dangerous game. I would even go so far as to say that other languages have become quite good at many of the things that Python is good at (namely pace of development) while being much better at the things that it’s not good at (performance, package management, etc), so I actually wouldn’t recommend starting new projects in Python except for certain niches, like scientific computing (and who knows if Python will even retain its dominance there).

I'm curious about sources for your claim of Python being 100-1000x slower than Go or Java. While the lower bound of the cited range is realistic (but only for certain problems), the majority of the range is highly unlikely. According to my sources, the difference is not that dramatic: depending on a specific type of a problem, it ranges from 1.5x to 100x. For applications using relevant Web frameworks, the composite difference is just 4-5x.

As for scientific computing domain, I would start a new project in Julia rather than Python.

It’s a ballpark estimate from various real world benchmarks I’ve done over the years (I’ve spent a lot of my career optimizing Python, including rewriting things on occasion). In the case of web framework benchmarks, I’m guessing that the fastest Python web frameworks are almost pure-C, and the benchmark is probably running very little actual Python code (while the Go benchmarks are pure-Go) this doesn’t extrapolate well to real world applications where the web server is never the bottleneck and your bottleneck tends to be app code that isn’t easily rewritten in C. Mostly the 100-1000X figure is intended to compare pure-Python code with pure-Go or pure-Java.

I guess, your ballpark estimate is too rough (and/or Python 3 has improved significantly since then). The sources / benchmarks that I used to back up my claims [1-3] imply use of pure Python, not "almost pure-C". That includes Web frameworks (FastAPI that I compared to Java / Go frameworks is pure Python, which, in turn, is based on Starlette, a pure-Python implementation of an ASGI framework).

[1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

[2] https://github.com/frol/completely-unscientific-benchmarks

[3] https://www.techempower.com/benchmarks/#section=data-r19&hw=...

Nobody would pick Python because it performs well. In many (and as performance of computers in general improves, more) cases, it performs well enough though. For one (admittedly absurd) example, where it just doesn't: a network device driver in user space: https://github.com/ixy-languages/ixy-languages

But if you fix Python package managers that will remove 50% of the audience for Docker. Think of the children! ;)

Definitely this, but I doubt such a luminary of computer science will interested in such work.

The XKCD comic on Python package managers/Python environments is not an exaggeration. I've always wanted to get more into Python but every time I attempt to, it's this hurdle that dissuades me.

Edit: Also, I guess Poetry is another thing that came along since my last attempt.

> The XKCD comic on Python package managers/Python environments is not an exaggeration.

Thanks, I haven't seen it before. It's really quite close to the state on my home machine:

https://xkcd.com/1987/

It's not that I couldn't eventually resolve the current state, it's that at the moment a few programs I regularly run work, in spite of the different dependencies they have, and I know that starting "cleaning up" will cost too much time.

> > The XKCD comic on Python package managers/Python environments is not an exaggeration.

> Thanks, I haven't seen it before. It's really quite close to the state on my home machine:

> https://xkcd.com/1987/

Hmm. I've had machines with similar states, though never with both Anaconda and Homebrew.

Instead, I often had (in addition to the rest of the system-level snarl) isolated per-app Python interpreters that were specified & constructed with Buildout for development testing & deployment.

Though I don't quite get the arrow from Homebrew 2.7 to Python.org 2.6. Is that actually a thing or hyperbole?

> Is that actually a thing or hyperbole?

It's fiction, of course. Art often has to exaggerate to make a point. I exaggerated too, just to remain in the artistic rather than the technical spirit of the comics. But there are some important bits of truth there, like in every good joke.

Of course. I just wasn't sure which bin that bit belonged to.

Is python the worst for package management?

I thought C and C++ might have similar issues since there's no unified package management there either.

So I hear Poetry is the way to go these days for python.

But a plurality of the people I encounter in the Clojure community came there because leiningen (Clojure's package manager that uses Maven under the covers) "just works" and they got tired of having a tough time reproducing builds consistently on other platforms / OSs with Python; not to mention the performance gains of the JVM.

Python's package management is light years behind, the much hated Maven.

I think in general Python's biggest challenge is that it doesn't scale well. This is an agglomeration of issues around the same theme: bad packaging when there's a lot of cross-cutting dependencies, slow performance, no concurrency, typing as second-class citizens, etc. All of that is barely noticeable when you're just getting started on a small experimental project, but incredibly painful in large production systems.

I strongly suspect that devs' satisfaction with Python is strongly correlated with the size of the codebase they're working on. Generally people using Python for one-off projects or self-contained tools tend to be pretty happy. People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language.

What I've observed a lot is that many startups or greenfield projects start with Python to get an MVP out the door as fast as possible. Then as the scope of the software expands they feel increasingly bogged down and trapped in the language.

And how much of those problems are an artefact of moving fast and getting things down.

I've seen the exact same scenario with other languages. The problem is that in a start up environment you are likely adding amd retiring more "features" at a speed that layers so much complexity that you can no longer reason about what business rules are actually valid any more.

I think that's part of it. There is a convention over configuration issue as well. A language like Go forces some patterns like package management and formatting unless you actively try to subvert it.

It wouldn't surprise me if many of these issues are self-selecting in the language communities as well.

No concurrency? asyncio is great for I/O bound network stuff!

"No parallelism" is probably what was meant.

> "No parallelism" is probably what was meant.

Which is still wrong, of course, but "no in-process (or in-single-runtime-instance) parallelism" would be correct, as would "forking inconvenient parallelism".

Posix fork() doesn't really count, if that's what you mean...

Why doesn't Python's multiprocessing module (which uses fork by default on Unix) count? It literally exists for parallelism.

It doesn't use fork() on macOS anymore, because some of Apple's own APIs get broken by its use.

Pretty much any app that uses both fork and threads, has to jump through many hoops to make the two work together well. And this applies to all the libraries that it uses, directly or indirectly - if any library spawns a thread and does some locking in it, you get all kinds of hard-to-debug deadlocks if you try to fork.

So unless you have very good perf reasons to need fork, I would strongly recommend multiprocessing.set_start_method("spawn") on all platforms. No obscure bugs, and it'll also behave the same everywhere, so things will be more portable. Code using multiprocessing that's written to rely on fork semantics can be very difficult to port later.

You wouldn't fork() for performance, but for security reasons.

It's understood that you can have "parallelism" by running two copies of your program using basic system facilities like fork(), or even by buying several computers and running one instance of your program on each of them. That's not what is meant by a language "supporting parallelism". If it was, then every language ever designed supports parallelism and so the term is meaningless.

To claim that a language "supports parallelism", it has to do something more to facilitate parallel programming. I would say that parallel threads of computation with shared memory and system resources is the bare minimum. You can go the extra mile and support transactional memory or other "nice" abstractions which make parallel programming easier.

Saying that Python support parallelism because it has a fork() wrapper is like saying that Posix shell is a strongly typed language because it has strings and string is a type.

It's not wrong. If running two processes counts as parallelism, then everything does parallelism, and it becomes pointless to talk about it.

Then one should talk about how convenient the related abstractions are. I like the concurrent.futures library.

Concurrency is not the same as parallelism. Python has good concurrency support, I agree. Python (C Python) does not support parallelism however due to its Big Interpreter Lock which actively prevents any parallelism in Python code.

This was probably a conscious design decision on the part of C Python implementers and perhaps a good one. But we should not claim that Python is something which (actively and by design) it's not.

I use `concurrent.futures.ProcessPoolExecutor` fairly often. I handle inter-process communication usually through a database or message queue, expecting that someday I'll want to go clustered instead of just single-machine multiprocessing. I've been burned by implementing multithreading and then needing to overhaul to clustered enough times to stop doing it.

Have you ever seen O(million) lines enterprise codebase that didn't suck?

LOL!!

Or even worse - could you imagine how many lines that would be in C++ ?

Yowza!

I've worked on several separate projects of that size in C++ and Go. None of them seemed to achieve a similar mess as Python codebases with one or two dozen thousand lines seem to. OTOH, all the typing developments in Python should have helped? I don't have that much experience with them in enterprise setting.

I have - and it's not that bad. The key is you have to have someone coordinating and driving a shared vision for the codebase and patterns. But it's hard to find people with that sort of passion and drive to follow-through as it's a multi-year endeavor with politics all over.

Otherwise its a thousand implementations of the same 100-line piece of code interspersed everywhere.

It seems like quality code management gets passed over by (bad) management because it looks like it doesn't directly move the project forward.

Which is strange because those same managers may be full adherents to micro tasking projects in a project management system whose purpose is basically to do for the project what code management does for the code itself.

In my workplace, we've recently had leadership that appreciates these things, and the difference is night & day. Simple requests from "stakeholders" (I hate that term) are often filled in days, or same day, instead of weeks. I think it helps tremendously that the primary manager is also a coder herself, and still codes ~25% of her job.

That's the problem with some languages - they lack a visionary, that drives the overall understanding of how things should be structured.

I believe it's Guido that basically said - if you don't like how Python does it, then implement it in C. And that's how you end up with great C based libraries bound to python... and python is often used as a messy orchestrating language.

This is surely anecdotic and very subjective, but I have (in Java and in C++; IIRC the exact versions were Java 7 and C++03), and the level of pain was lower than with a Python code base that was about one order of magnitude smaller. In the case of C++, the pain was mostly asociated with an ancient build system we used; the code itself was relatively manageable. There was almost zero template code, and maybe that helped (although in other occasions I've worked with smaller C++03 codebases that relied heavily on templates and I didn't find them that bad).

Not all codebases are equal and maybe I was lucky, but in my experience, using dynamic languages (or, to be exact, any language where the compiler doesn't nag you when there is a potential problem) doesn't scale well.

I've worked with an O(100k) line code base in Python that was pure torture. Honestly, I was so desperate for static-typing by the end that I would have preferred if it was all written in C++.

Large codebases are really hard reason about without types. I'm glad we now have projects like Pyre that are trying to bring typing to Python.

> desperate for static-typing

Can you explain why? I honestly don't know, because my experience with C++ was during school ~20 years ago, and since then professionally I've used mostly python in relatively small codebases where it's all my own code (mostly for data processing/analysis). Thanks!

(Although I did have to write some C code to glue together data in a very old legacy system that didn't support C++, much less python. It took a lot more effort to do something simple, but it was also strangely a really rewarding experience. Kind of similar to feeling when I work with assembly on hobby projects)

The main problem with duck-typing like python has is the lack of consistency between different objects that code has to work on. Different callers may pass objects with different sets of methods into a function and expect it to work. You run into the case where the object that was passed in is one with subtly-mismatched behavior from what your method expects, but you don't know who created it - it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on.

Static typing prevents that by telling you early where the mismatch is happening - some method calls into another with a variable of the wrong type, and that's where the bug is. It also allows tooling to look up the types of variables and quickly get information about their properties.

Got it, that makes sense. It also makes sense why I've not much been bothered by it in python since my relative small code bases don't have that many layers of abstraction laid on top of each other. I'm generally not working with more than 2,000-3,000 lines, and I can just about keep the basic structure in my head. (Unless it's been a while since I've had to revisit it... then I often hate my past self for getting "clever" in some way)

For these small code bases, static typing is still great (if you are used to it already) but the adverse effects of not having it usually show much stronger with a team (and not a single person). And yeah, if you keep the structure in your head, then you are good anyways.

> it was probably stored as a member variable by something 10 callstack levels and 5 classes distant from what you're currently working on

If you can define methods on an object dynamically in Python, it doesn't mean that you should. Monkeypatching is not encouraged in culturally in Python. Most often it is seen in tests otherwise, it is rare.

Nobody forbids using ABCs to define your custom interfaces or using type hints for readability/IDE support/linting (my order of preference).

There's some confounding issues that are often confused together tho.

large python code bases _could_ be written with well modularized, clean separation of concerns and composability. Or it could be written in spaghetti.

Using types _could_ help a code base from becoming spaghetti, but it's not the only way. I think the understandability and maintainability of a code base has more to do with the person writing it than the availability of a type system tbh.

No they can't, at least not with the same amount of effort. Of course, you can make everything good by throwing enough time and money on it, but that's not the point.

The issue is that to have a nice and well architected code base, you have to constantly refactor and improve - sometimes you need to re-arrange and refactor huge parts of the code. Without types _and_ tests, this is just not gonna happen. It will be unproductive and scary, so that people will start to stop touching existing code and work their way around it.

> I think the understandability and maintainability of a code base has more to do with the person writing it than the availability of a type system tbh.

That is the same thing. Because someone who wants great maintainability will also want a great type system (amongst other things).

A good carpenter never complains about his tools. He works around their limitations or uses something else.

The quality of the product is down to the skill of the worker either way.

We can assume buffer overflows are less common in Java than in C and I doubt that Java programmers are better craftsmen.

The same with types: they make some kind of errors much less likely though there is no silver bullet in general e.g., I much prefer a general purpose language such as Python for expressing complex requirements in tests over any type system (even if your type system is turing complete and you can express any requirement in it; it doesn't mean it is a good idea)

How often is a carpenter told to use this particular rusty saw or their work won't be compatible with everyone else's?

Everything interlocks in such intricate ways that you can't meaningfully choose your own tools, and working around problems only goes so far. And you can't repair your own tools.

There's also failures of the community to provide good guidance.

Funny, I'm working on a project about the same size, and the overly aggressive type and value restrictions are the main problem that I struggle with daily.

I've worked with Python for more than 15 years, usually with code bases 50-100k lines per app. The only time I have had real issues with types was a codebase I inherited where previous developers were using None, (), "", [], and {} all to mean roughly the same thing and sometimes checking for "", sometimes () etc. I couldn't handle it, so I put asserts everywhere and slowly found out where those things were coming from and sanitized it to be consistent.

I am literally working on two projects that are roughly 100kLOC each.

The Scala Spark project I can navigate, understand, test and consider to be average complexity... with some failures, unique to Scala.

The Python Spark project is barely readable.

People who built the Python Spark codebase are "experienced Python devs". While Scala codebase was built by people who used Scala for the first time.

(take this anecdote, as evidence for the poor tooling and guidance present in python community.... and BDFL's own failures)

I work on Python every day on a reasonably large code base and have none of the issues you’re talking about. I’m 10x more productive than similar C or Java projects.

Dependency management is about as easy as it is going to get. We have problems with our dependencies breaking stuff, but who doesn’t?

People talk as if packaging is a solved problem. It isn’t in any language. And then they complain that Python packaging changes too much. That’s because folks are iterating on a hard problem.

Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.

Curious to know what issues you have with deploying Python codebases. Out of all of the minor and major gripes I have with Python, deployment is not one of them.

To me, python deployments are painless, as long as you can stick to pure dependencies and possibly wheels.

Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting.

I've done that too much to like python anymore.

Yup, yup. I deploy statistical models with Python, and these always have C dependencies.

Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy.

I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.

Adding data files via setuptools....

And putting them into a common shared directory.

Try doing that without writing convoluted code in your setup.py.

It is funky, but importlib package resources helps.

Here's some of the ways to deploy Python code:

- `curl -L https://app.example.com/install | sh` that downloads installer and runs for instance: apt/yum install <your-package>

- in CI environment on a VM: `git checkout` & `pipenv install --deploy`

- `pipx install glances` on a home computer

- just `pip install` e.g., in a docker container [possibly in a virtualenv]. For pure Python packages, it can work even in Pythonista for iOS (iphone/ipad)

- just copy a python module/archive (PyInstaller and the like)

- give a link to a web app (deployed somewhere via e.g., git push)

- for education: there are python in the browser options e.g., brython, repl.it, trinket.io, pythontutor.com

- just write a snippet in my favourite editor for literate devops tasks/research (jupyter-emacs + tramp + Org Babel) or give a link to a Jupyter notebook

- a useful work can be done even in a REPL (e.g., Python as a powerful calculator)

The fact that you have 9 different ways all with their own different problems is exactly the problem here.

Do you use a single program on all of your devices for all possible computer-related tasks? Do you see a fault in such logic? Why do you think the deployment space is any different: do you use kubernetes for everything?

I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.

My core problem is with C/C++ depedencies. Can you describe to me how you handle these when you deploy Python?

  - system packages (deb/rpm/etc)
  - binary wheels (manylinux)
  - building from source

plus some caching if appropriate

God, I wish that would work for me.

To take your examples in order:

1) system packages: almost always out of date for my needs

2) Binary wheels: I actually haven't investigated this much, maybe it will work (and if it does, I'll buy you a drink if we ever meet in person).

3) Building from source: this kinda proves my point about Python having poor dependency management tools if this is a serious response. In general, this would be much further down the rabbit hole than I want to go.

> Do you use a single program on all of your devices for all possible computer-related tasks? Do you see a fault in such logic?

No. But if I talked about how I used 9 different word-processing programs, you'd see that as a problem, or at least an indictment of those programs. Deployment isn't that complicated.

> I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.

I use Maven/Scala and as far as I can see it covers all of them other than "give a link to a web app" which isn't actually deploying at all (and I'd still have used maven to deploy the webapp wherever I was deploying it).

I don't think there's any legitimate case for curl|sh, and I don't think there's any real reason for separate pip/pipenv/pipx (did you make that one up? Have I fallen for an elaborate troll?) - rather pipenv exists to work around only being able to install one version of a library at a time. Nothing's gained by having "just copy a module/archive" be different from what the tool does. Running in browser, notebook, or REPL can and should still use the same dependency management tooling as anything else.

If I want to deploy my code, I use maven. You can use curl (since maven repositories use standard HTTP(S)) or copy files around by hand, if you have a use case where you need to, but I can't think what that would be. If you want to bundle up your app as a single file, you can configure things to do that when publishing, but the dependency resolution, repository infrastructure, and deployment still look the same. Even if you want to build a platform-level executable, it's the same story, all the tooling just works the same. If I want a REPL or worksheet, I can start one from maven (and use the same dependency management etc. as always), or my IDE (where it's still hooked up to my maven configuration). If I want to use a Zeppelin notebook then there's maven integration there too.

Ever wonder why you don't hear endlessly about different ways of doing dependency management in non-Python ecosystems? Because we have tools that actually work, and get on with actually writing programs. It baffles me that Python keeps making new tools and keeps repeating the same mistakes over and over: non-reproducible dependency resolution, excessively tight integration between the language and the build tools, and tools and infrastructure that can't be reused locally.

I use Anaconda exclusively and deployments (with virtual environments) have been fairly ok.

That said, I do run into trouble when I have a dependency that requires compilation on Windows (i.e. like the popular turbodbc) because say, a wheel isn't available for a particular Python version. Any time a compilation is needed, it's a headache. Windows machines don't come with compilers, so one has to download and install a multigigabyte Visual Studio Build Essentials package just to compile. Sometimes the compilation fails for various reasons.

Require gcc compilation is headache for installing dependencies inside Docker containers too -- you have to install gcc in order to install Python dependencies and then remove gcc after.

I think requiring local compilation (instead of just delivering the binary) is a UNIX-mindset that is holding back many packaging solutions. I think a lot of pain would be alleviated if we could somehow mandate centralized wheel creation for all Python versions, otherwise the package manager marks a package as broken or unavailable and defaults to the last available wheel.

Also if only we applied some standards like R's CRAN repo does -- ie. if it doesn't pass error checks or doesn't build on certain architectures (institute a centralized CI/CD build pipeline in the package repo), it doesn't get published -- the Python packaging experience would be much improved.

Yeah, if PyPi was as annoying as Cran with respect to new versions, then a lot of this pain would go away.

For those who don't realise, when there's a new version of R, anything that doesn't build without errors/warnings is removed from the archive.

This is really annoying if you want something to keep running, but it prevents the kind of dependency rot common to Python (recently I found a dependency that was four years out of date).

Production deployment is Docker all the time.

Deployment for development is just pyenv and virtualenv.

This described is perfectly

At O(million), the problem wrangling it has more to do with how well its architected and written than it being Python. Python is at least easy to read. Its major deficiency is the lack of annotation of parameters, and that's something that could now be fixed... but it isn't going to be fixed in that much historical code.

It you are trying to get performance out of it (which doesn't really hinge on whether it's a million lines of code), then Python might be the wrong choice. But you can always write it in Rust or C and give Python an API to the functionality.

I agree that packaging is a mess. Fixing that mess with modularization in Java took a long time, and most other languages have that problem, too.

I disagree. Python is not inherently easy to read.

Explicitness and naming standards screw up the clarity of any code... Not to mention the complexity when you get into OOP.

Also lack of switch-case statements dont help. (Workaround is either if statements or dict of functions to be called)

"devop" here

> doesn't scale well.

Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points.

> bad packaging when there's a lot of cross-cutting dependencies

Much as I hate it, docker solves this. Failing that poetry or if you must venv. (if you're being "clever" statically compile everything and ship the whole environment, including the interpreter) its packaging is a joy compared to node. Even better, enforce standard environments, which stops all of this. One version of everything. you want to change it? best upgrade it for everyone else.

> slow performance

Meh, again depends on your use case. If you're really into performance then dump out to C/C++ and pybind it. fronting performance critical code in python is a fairly decent way to allow non specialists handle and interface performance critical code. Its far cheaper to staff it that way too. standard python programmers are cheaper than performance experts.

If we are being realistic, most of the time 80% of python programs are spend waiting on network.

Granted, python is not overly fast, but then most of the time your bottleneck is the developer not the language.

> no concurrency

Yes, this is a pain. I would really like some non GIL based threading. However its not really been that much of a problem. multiprocessing Queues are useful here, if limited. Failing that, make more processes and use an rpc system.

> typing as second-class citizens

The annotation is under developed. being reliant on dataclass libraries to enforce typing is a bit poop.

> People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language.

I work with a _massive_ monorepo. Python isn't the problem, its programmer being "clever" or making needless abstractions of abstractions. None of that is python's issues, its egotistical programmer not wanting to read other people's (un documented) code. And not wanting to spend time make other people's code better.

>Nothing scales well. scaling requires lots of effort. It doesn't matter what language you use, you'll rapidly find all its pain points.

This is very important. A lot of people think that just using go or rust or whatever other language is new fixes all of this. But with a big enough project, you'll find all the issues. It's just a matter of time.

Do not miss that one will find all of the language's pain points. I'd wager that a dynamically typed languages such as Python has quit a few more pain points at scale than a more principled language such as OCaml.

I love Python's bignum arithmetic when I write small prototypes for public key cryptography. I love Python's extensive standard library when I'm scrapping a couple web pages for easier local reading. But I would never willingly chose it for anything bigger than a few hundred lines. I'm simply not capable of dealing with large dynamically typed programs.

Now if people try Rust or OCaml with the mentality of an early startup's Lisp developer, they're going to get hurt right away ("fighting the language" and "pleasing the compiler" is neither pleasing nor productive), and they're going to get hurt in the long run (once you've worked around the language's annoying checks, you won't reap as much benefit).

If you'll allow the caricature, don't force Coq down Alan Kay's throat, and don't torture Edsger Dijkstra with TCL.

Though OCaml’s tooling pain points hurt at least as much as Pythons, even though I adore the language.

This is somewhat true - scaling is hard no matter what - but some things scale much better than others. I have been miserable working with ruby on rails codebases that are much smaller than java codebases I have been content working on. This is despite personally enjoying the ruby language far more than the java language.

> Much as I hate it, docker solves this. Failing that poetry or if you must venv. (if you're being "clever" statically compile everything and ship the whole environment, including the interpreter) its packaging is a joy compared to node. Even better, enforce standard environments, which stops all of this. One version of everything. you want to change it? best upgrade it for everyone else.

No, docker doesn't solve the fact that some packages just won't play nicely together. NPM actually does this better than the python ecosystem too since it will still work with different versions of the same dependency. You get larger bundle sizes but that's better than the alternative of it just flat not working.

its programmer being "clever"

When I have to revisit old code I've written, I occasionally encounter my "cleverness" at the time. I always hate that past version of me. I think I've mostly learned my lesson. I guess I'll know in a few years.

"When I have to revisit old code I've written, I occasionally encounter my "cleverness" at the time. I always hate that past version of me. I think I've mostly learned my lesson. I guess I'll know in a few years."

...I feel attacked.

I'm sorry, but as a fellow Python "devop" too, this really reads like empty apologism.

>Nothing scales well. scaling requires lots of effort.

Sure, just like all PLs have their flaws, and most software has security vulnerabilities. But it's a question of degree and the tendency of the language. Different languages work better in different domains, and fail in others, and what Python is specifically bad at is scaling.

If only for the lack of (strong/static) typing and the relatively underpowered control flow mechanisms (e.g. Python often using exceptions in their stead)... While surely all languages have pain points that show up at scale, Python still has a notable lot of significant ones precisely in this area.

>docker, poetry, venv...

Yes, and this is exactly the point. There's at least three different complex solutions, none of which can really be considered a "go-to" choice. What is Rust doing differently? Hell, what are Linux distros doing differently?

>If you're really into performance then dump out to C/C++ and pybind it.

If you want performance, don't use Python - was the parent's point.

>If we are being realistic, most of the time 80% of python programs are spend waiting on network.

This really, really doesn't apply to all of programming (or even those domains Python is used in). Besides, what argument is that? If it were true for your workload, then it would be so for all other languages too, meaning discussion or caring about performance is practically meaningless.

>Granted, python is not overly fast, but then most of the time your bottleneck is the developer not the language.

Once again, this applies to all languages equally, yet, for example, Python web frameworks regularly score near the bottom of all benchmarks. I doubt it is because of the lack of bright programmers working in Python, or the lack of efforts to make the frameworks faster.

>Python isn't the problem, its programmer being "clever" or making needless abstractions of abstractions.

Just as C isn't the problem, it's the programmer forgetting to check for the size of the buffer, and PHP isn't the problem, it's the programmer not using the correct function for random number generation.

You can always trace any given error to a single individual making an honest mistake, that's really not a useful way to think about this. It's about a programming language (or an environment) leading the programmer into wrong directions, and the lack of safety measures for misguided "egotistical programmers" to do damage. You can blame the programmers all you want, but at the end of the day, the one commonality is the language.

Now Python is still one of my favorite languages, and I think that for a lot of domains, it really is the right choice, and I can't imagine doing my work without it. But performance and large, complex systems, is not one of those domains, and I honestly feel like all you've said in Python's favor is that other languages are like that too, and that it's the fault of the programmers anyway.

I have thought about what you've written. I broadly agree. I didn't mean for my post to be a "python is great really", It was more to illustrate that all programming languages have drawbacks.

The is a points that I think I've failed to get over:

> Just as C isn't the problem, it's the programmer forgetting to check for the size of the buffer, and PHP isn't the problem, it's the programmer not using the correct function for random number generation

I don't think I was arguing that point. of course all languages have their USP. My point I wanted to get across is that large python projects are not inherently hard to manage. That kind of scaling is really not that much of an issue. I've worked on large repos for C, C++, python, perl, node and as a punishment, php. The only language that had an issue with a large codebase was node, because it was impossible to build and manage security. The "solution" to that was to have thousands of repos hiding in github.

The biggest impediment to growth was people refusing to read code, followed swiftly by pointless abstractions. This lead to silly situations where there were 7-12(!) wrappers for s3 functions. none of them had documentation and only one had test coverage.

Scalability is not just runtime, it's also developer time scalability. The larger the project, the more you have to split it up and write interface documentation between libraries - which adds complexity.

As for processing scalability - Python is OK, but it's considerably hampered by BDFL's own opinions. The result is a few third party libraries that implement parallelism in their own way. That functionality should be integral to the standard library already. The worst part is lack of standard API for data sharing between processes.

> packaging

Python's packaging issues only start with package management. Setuptools is a wholly mess of a system, that literally gave me headaches for the lack of "expected features". I hate it with every single cell in my body.

And then there are systems and libraries, where you literally cannot use docker (Hello PySpark!).

>read other people's (un documented) code

I lolled! Seriously... We get Python fanboys moan about how indentation makes everything more readable and it's a pleasure to write code in python. Give me a break!

>People stuck in sprawling enterprise codebases, with O(million) lines of code to wrangle, seem almost universally miserable with the language.

This seems to be the case with most languages, especially if good code control isn't practiced, and unfortunately that's not uncommon.

Very much agree. I oversee a relatively small python codebase, but getting good quality, safe code out of the developers in a controlled way is really hard - there are so many ways in which the language just doesn't have enough power to serve the needs of more complex apps. We have massive amounts of linting, type hinting, code reviews spotting obvious errors that would be just invalid code in other languages.

It's like getting on a roller coaster without a seat belt or a guard rail. It's fun at first, and you will make it around the first few bends OK ... then get ready ...

Of course, with enormous discipline, skill and effort you can overcome all this. But it just leaves the question - really, is this the best tool for the job in the end? Especially when you are paying for it with horrifically bad performance and other limitations.

I work at Instagram, which is a O(millions) LOC Python monorepo with massive throughput and a large engineering team. It's actually quite nice — but our code is heavily, heavily typed. It would be miserable without the type system. Some of the older parts of the codebase are more loosely typed (although they're shrinking reasonably quickly), and those sections are indeed a huge pain to ramp up on.

Part of the success of IG's large Python codebase is FB's investment into developer tooling; for example, FB wrote (and open-sourced, FWIW) our own PEP-484 compliant type checker, Pyre [1], because mypy was too slow for a codebase of our size.

1: https://github.com/facebook/pyre-check

Have you guys published any whitepaper on this subject? These last few years working on moderately large python codebases with dynamic typing have been less than idilic.

Here's a small collection https://pyre-check.org/docs/pysa-additional-resources

Yeah, you could use my hairline to tell when I’m working with Python... I still haven’t figured out a good typed framework for it yet.

I also struggle with weak(er) typing in Python, compared to strong(er) typing in C++, Java, or C# -- or even VBA(!).

Frustrated like you, I wrote my own open source type checking library for Python. You might be interested to read about it here: https://github.com/kevinarpe/kevinarpe-rambutan3/blob/master...

I have used that library on multiple projects for my job. It makes the code run about 50% slower, on average, because all the type checking is done at run-time. I am OK with the slow down because I don't use Python when I need speed. My "developer speed" was greatly improved with stricter types.

Finally, this isn't the first time I wrote a type checking library/framework. I did the same for Perl more than 10yrs ago. Unfortunately, that code is proprietary, so not open source. :( The abstract ideas were very similar. Our team was so frustrated with legacy Perl code, so I wrote a type checking library, and we slowly applied it to the code base. About two years later, it was much less painful!

Try using Pyre! It's open-source. I use it daily at IG.

That's my major complaint about Python...

For it's age and popularity - the tooling is abysmal.

I have to rewrite too many things, that I expected to just be there for the age of the project.

And some things were only fixed now! dict + dict only started to work with 3.9!

dict1 + dict2 is syntactic sugar, you had been able to do `dict1.update(dict2)` since forever.

No, it's completely different: dict1+dict2 creates a new object and leaves inputs unchanged, dict1.update(dict2) modifies dict1.

you've been able to do splat for forever! https://dpaste.org/yU7s

Could I ask you what language you use instead then?

Is concurrency really an issue? Yes, you do not have threads, but you can launch multiple processes. Do you really need to habe shared memory for your concurrent needs (i think it is muuuch easier to introduce subtle bug into a shared memory concurrency (threads))

Poetry is very good: https://python-poetry.org/

It should be heavily promoted as the official option and included with the distribution.

Cargo for Python… that's pretty much Poetry.

I was already in love with Python, but Poetry has definitely made me a much happier Python developer.

I've never used anything I liked better than Maven Central + Gradle. Why can't everyone just copy the Java ecosystem when it comes to package management.

(Although the maven package ecosystem seems ideal to me, Gradle is just "good enough" - it's mostly the standards around versioning and tooling dealing with versioning and everything being there that makes it good to me).

Maybe they should just adopt the Nix package manager, and put some of Microsoft’s resources into making it work on Windows.

> sponsor it through Microsoft $$$

You don't want that. When companies "sponsor" things they try to take them over, unless it's a pure donation, which is rare. The community then drops out because a company is in control. Later the project is abandoned by then company. It's a slow death spiral.

I would love if Guido could create a new PEP for extending modules with generic namespaces, ala Perl/CPAN modules.

There aren't 15 different libraries for doing the same thing in Perl, there's 1. You never replace it, you extend it by making a new module in a hierarchical namespace. The same core library's code might not change in years while new extensions can keep popping up. So even if you think Requests sucks, you can make Requests::UserAgent which inherits Requests code and extends it / gives a better interface. And these can be written & packaged by completely different authors.

Then maybe Pypi wouldn't have 5,000 nearly identical yet mostly unusable modules, or modules with nonsense names.

I had a job interview today and said this exact last sentence ahah

Just use poetry (my preference) or pipenv. This has been a solved problem for a long time.

What’s wrong with virtualenv + requirements.txt + pip?

What's wrong with poetry right now?

defaulting to installing and finding packages locally would be a big win for me.

Gnu Guix would be a great one ! (the package manager, not the distro GNU Guix System)

https://guix.gnu.org/

https://guix.gnu.org/en/blog/2019/gnu-guix-1.0.0-released/

https://guix.gnu.org/en/blog/2020/gnu-guix-1.1.0-released/

https://guix.gnu.org/#guix-in-other-distros

https://guix.gnu.org/en/blog/2020/guile-3-and-guix/

I'm hoping for a Jupyter/Excel hybrid, but that's mostly wishful thinking.

Please no. Excel is great when it stays within its lane and use case and doesn't try to be everything. Jupyter is okay-ish is some places but in general is way overused. Mixing them together would be a move in the wrong direction and a bit of a mess.

Oh no.

If you mean to retire VB in favor of Python for writing macros, I'm all for it. Not much Jupyter though

That's arguably the goal of [Pluto.jl](https://github.com/fonsp/Pluto.jl). It's a (Julia-specific) reactive notebook that automatically updates all affected cells and has no hidden state.

Ooh that's really good, thank you for sharing!

That looks so good that I am thinking about learning Julia.

Well then I would like to through this your way. https://mitmath.github.io/18S191/Fall20/

Thx I will check that.

It's already been done https://www.youtube.com/watch?v=Od-1cEac9U4&feature=emb_titl...

There's been a few projects in this space (PySpread is the first that jumps to mind), but also, not too long ago (last year maybe?), MS was investigating making Python a first-class inhabitant of Excel, so might already be in the pipeline.

If I remember correctly, Guido resigned as BDFL in 2018.

This is good news. It's sad that I find a lot of interesting possibilities on Azure end up being "windows only." Hopefully having more support for Python will fix some of that.

I'm not sure that Python will profit form this. He has had 50% positions before and most of the new things he did were precisely in the period in which he did not work for a corporation.

If anything, this further corporate influence on Python development is not something to be applauded. I bet there will be lots of more churn in the next five years, big announcements and no results.

I hope they use Microsoft's money to make Python fast (or at least fast enough)

At the top of my wish list there is a deployment tool for web applications like capistrano / mina for Ruby. I ended up writing my own for the project of a customer. It's been running for a few years now.

Same customer, another project, I'm experimenting with a deployment system based on git format-patch. It copies the patches on the server (we have only one server) and applies them with patch. Then restart the web app.

It's fun to learn the internals by rewriting the tooling but a good tooling to start with would be better.

Solve "How to deploy my sweet Python script on grandma's computer" problem.

That is some reasonable way to deploy to someone who is not a programmer nor tech person.

Not sure how much has changed since this was written: http://effbot.org/pyfaq/can-python-be-compiled-to-machine-co...

PyInstaller.. well.. it works okay with std.. sometimes it does not. It is the Electron solution.. add pandas to a project and you get a 600MB install.

As developers we can manage to deal with package managers.

Such a shame things like that used to be simple - Py2Exe was fine and Windows didn't freak the hell out at the sight of an unsigned binary.

In the education space (where Python is pretty big), the biggest problem we have is that Python code isn't 'shareable' so it's hard for kids to show off their creations to Grandma. Sure, there are some products which work to work around that (like repl.it), but the core problem remains - a student can't easily get their Python program to run on an arbitrary computer. It's why I'm very tempted to move to JS for teaching, despite all of the dragons lurking there.

I agree, it’s one of the reasons I stopped teaching my kids Python and switched to JS. It is so easy to share and so portable, and runs everywhere!

Not just to grandma. I have trouble deploying my scripts to my colleagues.

I had this with a collleague the other day:

1. Install foo

2. No not that foo!

3. Reinstall python

4. Goto 1

I did that to myself so many times...

>Congrats to him for finding something fun to do in retirement - dictators usually end up with a different outcome. ;)

I guess he is going for a Diocletian cabbage farmer style retirement.

Why dictator? lmao I'm not familiar with the guy on anything else than he's PHP's creator

In the off chance this isn't a joke: Python, not PHP. And he wasn't just its creator, he was the head of the project for its whole life until his recent retirement. Typical term in the OSS community for one guy who is in charge of a massive OSS project is Benevolent Dictator For Life, or BDFL. And Guido was definitely one of those.

Does that mean I will need to switch from mypy to pyright?

I wonder if they asked him algorithm problems in his interview /s

This is cool though. I'll be curious to see what he works on.

I am pretty sure they did, unless he got hired at some VP/Distinguished Engineer level.

And to be clear: If you are unable to solve these common algorithmic questions that companies like Microsoft ask, then that's not the right place for you to work. This is a tangent, but there are literally thousands of companies that won't require you to solve these problems. The thing is, at Microsoft & co. you don't just do this in an interview. You do it at your job too. We do foundational work in many teams and we need to solve algorithmic problems practically every week. If you are unable to code yourself out of a DP problem or scared of NP completeness and approximation algorithms, then maybe find a different job instead of complaining about the interview process?

You really don't do it at your job.

I’m taking a cheap shot here, but the people maintaining Skype at MS certainly don’t do it. And there are such examples at other big companies.

He was hired as a Distinguished Engineer.

Yes, they hired him at Distinguished Engineer level.

> unless he got hired at some VP/Distinguished Engineer level.

Google or Wikipedia can tell you who Guido van Rossum is.

I believe the "unless" was about the position he was hired for, not about who he is.

Everybody needs to invert B-trees before breakfast

I wholeheartedly disagree.

1. This is Guido van Rossum. If I were him and asked to solve puzzles, I'd tell the hiring company to fuck off.

2. These quizzes aren't so bad, but the pressure and stakes make it incredibly stressful. There's no standard, and often times the interviewer is the one that sucks.

> We do foundational work in many teams and we need to solve algorithmic problems practically every week. If you are unable to code yourself out of a DP problem or scared of NP completeness and approximation algorithms, then maybe find a different job instead of complaining about the interview process?

I'm pretty sure your opinion here is not that of your employer.

Counterpoint:

When I was leaving Google the first time, I asked my skip lead (who was employee #48 there, ended up running all of Search, and was previously a core HotSpot engineer at Sun) why he chose to work at a small startup when, coming off of HotSpot in 1999, he could work anywhere. He replied "Aside from them being one of very few companies with an engineer-centric culture, they were the only company that required I interview. Everybody else was willing to hire me on the spot."

For some personality types - and particularly the ones likely to do world-class work - being challenged is a positive sign. It means that the employer does their due diligence, and they will mostly be working with other people who react positively to a challenge.

> It means that the employer does their due diligence, and they will mostly be working with other people who react positively to a challenge.

To me, due diligence would be more like using software that someone has created. If it feels snappy then they're good enough at algorithms for the kind of software that they create, if it doesn't then maybe it's worth looking into whether or not there's a good reason for that.

Like if you apply for a job at the NYT, I doubt they make you do a timed writing test with people staring at you and asking you questions in the middle. They probably just read some of the previous work you've done.

Are you kidding? They'd make Ansel Adams do a timed test on how to use Elements to do colour correction.

Sure, but due diligence doesn't mean asking Guido van Rossum - who's got bona fides coming out the wazzoo - to solve a whiteboarding problem.

How he solves the problem doesn’t matter. You don’t care in the interview if he had the answer memorized or if he fumbles through it.

I do not want to work with anyone who finds that getting their hands dirty is beneath them. It is very, very rarely going to be a good use of their time to do those problems. It will often be a good use of their time to teach those problems. A senior engineer, even one who’s unlikely to work with junior engineers on a regular basis, will need to explain their thinking. They need to show humility and compassion. Those are practiced attributes. This precise situation is the best practice you can get - New and Unknown person, some amount of challenge and complexity involved.

Thinking that whiteboarding problems are a bad use of time is a very strong signal for a senior person who is out of touch.

At an old job, my boss was moving desks and he came across an extra copy of CLR "Introduction to Algorithms" and he asked if anyone wanted it. As he was my direct supervisor, he said he'd give it to me only if I promised never to open it, and only to use it as a monitor stand.

I can see that. When I interview, I often compare the difficulty of the questions that different companies. I have noticed that I feel a little more respect to the companies that ask the more challenging technical questions (not puzzles) vs. the ones that ask the super basic ones. It does make me think that the ones asking the simpler questions are likely getting lower quality candidates and that I would be joining them.

When I left uni, I got around 15 job offers. I went with the one with the lowest pay, because that's where I had go to through the most difficult interview process.

(Unfortunately this happened in a small Eastern-European country, so the company was an investment bank, not Google.)

Well, he wanted to meet the team and have an idea who/what he would be working with.

It's really scary to go to company that is willing to hire you without ever talking to you.

Counter-counter-point. An engineering interview has a non-neglible amount of randomness. Maybe you get a grumpy interviewer or a noob interview, or your brain freezes over.

When you are considering hiring a nobody, this is acceptable. You will interview multiple people, and they will interview at multiple places, so the randomness isn't that important.

But if you want to hire one specific guy as a strategic hire, suddenly the randomness may no longer be acceptable.

> This is Guido van Rossum. If I were him and asked to solve puzzles, I'd tell the hiring company to fuck off.

Not sure what you're trying to get at:

MacOS homebrew creator is an effin nobody compared to Guido, therefore he should "know his place", "get in line" and invert a binary tree on the whiteboard and act like an obedient tech interview candidate that he really is?

MacOS homebrew creator should've told Google to fuck off?

I love Homebrew – but it really isn't that much of an achievement compared to damn Python, and I also have no idea who is its creator.

Homebrew is 11 years old. I'm willing to bet there are as many people (likely fewer) people who knew Guido in 2002, when Python was 11 years old, or even 2005, when Google hired Guido.

And I'm willing to bet when Google hired Guido in 2005, they didn't put him through a coding challenge humiliation clown show day.

Imagine a University asking a Physics Nobel Laureate to solve QM Problems from an undergrad textbook in order to get hired as a Professor. It would the height of lunacy and incredibly insulting.

Guido is not a Physics Nobel Laureate. Going with the Physics analogy, Python is more like an overgrown masters level project, not a Nobel prize level by far. He did a good job at growing the Python community, and this is a great achievement! It requires certain personal traits not everyone has. But at the technical level, he made many beginners mistakes when designing Python, which he tried to fix later, but not always successfully.

An overgrown masters level project, eh? You could probably say the same thing about the founding of the United States!

"The US constitution is like an overgrown enlightenment dissertation. The founding fathers did a good job at growing the United States, and this is a great achievement! It requires certain personal traits not everyone has. But at a technical level, they made many beginner's mistakes when drafting the constitution, which the country tried to fix later, but not always successfully."

:-P

Maybe.

It may come to you as a surprise, but for an outside observer, who hasn't been indoctrinated at school by the religion of American exceptionalism, the US might not be a very good example. Think of American military-industrial complex that apparently defines the country's foreign policy.

My analogy was meant to cut both ways, and serves as much as a criticism of the US constitution as it does a compliment!

You'd certainly be wrong to assume that Americans are all in favor of our foreign policy, to say the least. Aside from that, some of the downsides to the US constitution that inspired my comment include the way the founding fathers totally failed to anticipate that the nation would be completely polarized by a two party system, and that this polarization would happen along geographic lines.

Admittedly, this compromise whereby rural states with lower populations enjoy disproportionate political representation is baked into the constitution being agreed to in the first place (the 3/5ths compromise being relevant here as well). As we've moved to more direct democracy, with things like the electoral college being bound to the (local) popular vote and the direct election of senators, the original intentions of having a federation of mostly autonomous states becomes more and more anachronistic, while still fueling an increasingly polarized electorate that pits high-tax revenue and high population centers like SF and NY against low-tax revenue and low population centers that make up most of the country.

The United States is large geographic region with a heterogeneous economy. I don't know if a parliamentary form of government would have served this kind of country well, but certainly most friends from abroad who have spoken to me on the subject have implied that proportional representation is far more sensible than FPTP voting, and that parliamentary forms of government avoid the gridlock and polarization that our de facto two-party system engenders.

>Python is more like an overgrown masters level project, not a Nobel prize level by far.

Please find me another "Masters Project" that has 8 Million+ Users and drives the backend for thousands of tech companies worldwide. Good Luck.

oh friend, say no more, how about left-pad? https://www.npmjs.com/package/left-pad

Funny guy.

If you think a single function is equivalent to an entire Language you should get your head checked.

The number of users is not a metric that makes something Nobel prize-worthy. The exaggeration of the comparison with left-pad is to make this point more clear to you ~~ if you cannot see it yourself, it's sad. Also, going with "funny guy" and "check your head" is not a good argument btw, if you did not know that ~~

> MacOS homebrew creator should've told Google to fuck off?

Yes, that's exactly what he should have done.

I think the best experiment would have been to have him apply blind where the interviewers did not know he was Guido -- and see how he fared on the technical interviews.

> I think the best experiment would have been to have him apply blind where the interviewers did not know he was Guido -- and see how he fared on the technical interviews.

That would be amusing, but he'd have to be disguised, as he's rather recognizable, having done a lot of 'State of the Python' talks and the like.

He doesn't look as much like Rick Moranis as he used to, though, so that helps: https://gvanrossum.github.io/images/Guido@200dpi.jpg

From his twitter profile:

""" Python's BDFL-emeritus, Distinguished Engineer at Microsoft, Computer History Fellow. Opinions are my own. He/him. """

"Hello Guido thanks for coming in, we would like you to open up visual studio code and create a sudoku solver that can solve this partially filled out board"

Time to use prolog to get your revenge on the interviewers.

A Sudoku Solver in APL https://www.youtube.com/watch?v=DmT80OseAGs

More appropriately, in Prolog (by Markus Triska) https://youtube.com/watch?v=5KUdEZTu06o

Very nice, and I can understand this one.

Thanks for this link. I took a course on logic programming in school and it made a big impression. I found Prolog to be pretty mindblowing at the time and I'm happy to see it still is.

I think this was in response to this:

https://twitter.com/mxcl/status/608682016205344768

His Twitter actually says "Distinguished Engineer at Microsoft"

This comment really just drives home the nail of how awful the state of interviewing, and especially the mental state of some interviewers, in this industry with a backhoe bucket.

I really just want to thank you for putting this useless mentality on display.

I know it's not Microsoft, but D. E. Shaw asked Larry Summers math puzzles when he interviewed there. At the time, he was the president of Harvard University.

I think it's my favorite example of how crazy some interview processes can get, lol.

I’m not sure what’s crazier to me: asking someone to demonstrate a live proficiency of an abstract skill that is only tangentially related to that actual day-to-day activities of a role or just assuming that because someone has some high credential that they would be good in a given role.

DE Shaw is a financial company filled with maths guys, doing statistical analysis and modeling all day. A math puzzle is the most normal question you could be asked there.

The real question is why Larry Summers is going to a quant interview? Did he apply for a quant role?

They hired him as a managing director.

For many roles at a hedge fund, being able to do mathematics quickly and intuitively is a valuable skill. Not sure why an economist applying for an MD job would need to be tested on that, though.

He should have called their bluff and failed the quiz on purpose. What are they going to do?

Next time there's a tech interview discussion and someone defends it, linking this thread will be very useful.

Almost 30 years of BDFL of Python, sorry don't care, go do 200 leetcode before talking to us. And if you don't spit out the answer a few seconds faster than that fresh graduate, clearly you're a lesser engineer and should be rejected.

Wow, so much salt. It was a joke.

You should have left "/s" at the end of your comment, it's really hard to distinguish between a joke and stupidity on the Internet.

To me this undermines any humour the comment may had to begin with.

They did add a /s to the end. The user you were replying to wrote the comment that the stupid comment was replying to!

It is there. At the end of the line.

In that case, it was an unnecessary joke and not helpful to the conversation.

Sorry, I'll refrain from humor from now on?

Seems reasonable.

Go to reddit if you want to make average IQ humor.

Please don't, keep being fun!

It works here on HN too, many of my comments are upvoted jokes and I liked yours. It's sad it attracted animosity though.

I like browsing HN mostly for the interesting discussions, but I enthusiastically take the occasional jokes that come with them.

(I might have been upvoted by people taking my jokes seriously now that I think about it. I don't know if this is a terrifying or a funny thought!)

You must be injecting crack directly into your frontal lobe if you think that the creator of the world's third most important programming language being asked an algorithms question (unlikely) and failing it (entirely possible) means that he's unqualified to do "foundational work."

I had a Nobel Prize winner as a physics professor in college who got three successive different wrong answers when attempting a freshman physics problem in office hours. That doesn't mean that physics isn't the right place for him to work.

> third most important programming language

not going to debate that, but I'm curious, what do you rank as #1 and #2?

The tiobe index lists Java and C in those spots.

That's fair, I guess.

Python is number 2 now, according to https://www.techrepublic.com/article/python-overtakes-java-t... :)

We should not give too much importance to this, but the fact is that Python is now ranked consistently #1, #2 or #3 in sufficiently many rankings to consider it seriously.

> The tiobe index lists Java and C in those spots.

That list is ranking languages by current popularity, though, rather than 'importance'.

I'd define 'importance' as more along the lines of "amount of havoc created if all software written using that language were broken/deleted at once".

Wouldn't languages like COBOL rank pretty highly because a lot of legacy banking systems are written in that?

Yes, but frankly, C would still cause more havoc because even if the mainframes somehow continued to run, they would still be effectively inaccessible over any network.

>I am pretty sure they did, unless he got hired at some VP/Distinguished Engineer level

No way they even considered doing it... That's for people out of college...

Thats certainly invalid. People with 20 years of experience or more, are asked questions based upon the role they are interviewing for.

If you are applying for a principal or higher engineer role and your job involves coding, you are asked coding questions. May be not just focused on a complex bookish algorithm only, but rather more close to a real life distributed programming / synchronization problem etc. for example.

I know we're talking about FANG itw but come on you really think people like Guido needs to do any of that? They probably don't do "regular" itw, they go to a restaurant with some important people and that seels the deal.

Why the fuck would CS graduates be afraid of DP problems, NP-completeness or approximation algos??? That’s literally the table stakes of our profession.

The problem is rewarding rote memorization in a whiteboard interview, at the expense of actual understanding, and ability to research the problem.

> that’s literally the table stakes of our profession

Table stakes for what exactly?

Computer science graduates...? Like I said in the comment.

For sudo scientists.

> Why the fuck would CS graduates be afraid of DP problems, NP-completeness or approximation algos??? That’s literally the table stakes of our profession.

If the profession being discussed is "academic work in Computer Science", sure.

If it is "software development", those things absolutely are not really "table stakes".

If it is "software engineering", then I don't think there is broad consensus on what that profession even is, much less what table stakes in it are.

Are you seriously claiming Guido van Rossum was hired for software development/engineering..? I don’t understand the context switching here.

> Are you seriously claiming Guido van Rossum was hired for software development/engineering..?

No, nor do I think he was he hired for the other thing I discussed, academic computer science. The post I was responding to made a general comment about "CS graduates" and "our profession"; I was responding to that. Whether that post itself was material to, or merely tangential to, the discussion of GvR's hiring at Microsoft is an argument that, while perhaps interesting to some, was not the focus or concern of my response.

My post was a response to a specific comment, please don’t take it out of context.

Because these problems can have some subtleties that are hard to get right in a high-pressure environment. Some interviewers will completely write you off for small mistakes.

That sounds interesting! Do you have any examples?

I would have thought that if you actually needed people to perform under pressure, you would design your test explicitly around that, instead of using “comfort with the whiteboard” as a proxy...

That might be a fair statement for some roles, but what about the second part where you code it up and write the code on a whiteboard w/o running/debugging and have to get it right in 45min?

I’m not sure I understand..?

Do you mean it should be used as an actual speed test?

> I am pretty sure they did

Lol! I'd bet my year salary they didn't.

Well, you might want to check guido's twitter before making that bet.

What exactly am I supposed to check in his Twitter?

Whoops, mismatched your reply to the wrong branch of the comment tree

Every body is asked some thing close enough. At higher levels, often focus is not on coding, but enough depth of design where a person of such profile might end up educating the interviewer - while satisfying their requirements.

That being said, ability to solve coding problems efficiently (not necessarily spit A* graph algo in sleep), but a decent close to real life coding challenge is fair game.

> NP completeness

Programmer here. It's true. I deal with NP completeness every day.

Hi, I'm a programmer for Microsoft. I didn't have to answer silly algorithm questions for them to hire me. I'm nowhere close to VP/Distinguished Engineer level

(that said, my path to being hired did involve writing a sudoku solver, but that wasn't in an interview for a position at Microsoft)

Perhaps you could share your story to show that not all is dire like HN would make it seem.

Every story is unique. I was responding to a comment that was making sweeping claims

My story likely isn't replicable, but everyone has to find their way

I interviewed for Citus a couple weeks before they announced being acquired. I found out about the acquisition on Hacker News before having received an offer. They were able to get me in without going through the hiring process again

Initially I'd be moving to San Francisco but I wasn't eligible for any visas as I don't have a post secondary education. Staying in Canada's worked out

Some examples of things working out here: I was asked to implement some parsing, & well that's not so hard when you've written a Lua parser a year beforehand: https://github.com/serprex/luwa/blob/master/rt/astgen.lua (an astute reader will notice I don't handle precedence here, which is pretty important for arithmetic parsing. That's because I opted to implement shunting yard during codegen phase)

What value does this story give a reader? I only think it'd serve to continue an argument about how not everyone is so lucky, which isn't the original argument of "unless you're VP/Distinguished Engineer, not even Guido van Rossum gets to skip whiteboarding"

>We do foundational work in many teams and we need to solve algorithmic problems practically every week.

Keep telling yourself that as you fix mindless bugs in some Advertising platform lol.

> The thing is, at Microsoft & co. you don't just do this in an interview. You do it at your job too.

Oh so that's why the Azure Portal UI is such garbage. Their frontend developers are just busy solving knapsack problems...

There’s plenty of issues around algorithmic questions, and it’s even possible he would fail them, but there’s still important value in asking them.

Senior engineers in such companies need to work and cooperate with hundreds of people. If they find solving such problems beneath them, or expect to be treated as a higher class of person, or not willing to get hands dirty, they will not be as valuable as employees. (Occasionally, the lone genius is great, but the lone genius is a worse employee than the genius that can cooperate well)

My company would keep doing such interviews, but the expectations change of course. A new grad needs to do these well. More senior people can do worse, and make up for it using other skills showcased in other interviews, or based on their record.

But if you find a senior person that finds solving algorithmic questions is beneath them, you might want to be careful about them and how they will interact with your company’s culture.

I never remember the nitty-gritty details of the more complex algorithms and data structures, I only remember they exist and their important properties. If I then have to implement one, I consult google or books. Googling often has the neat side effect that you'll sometimes find refinements/improvements or other (new) potentially better algorithms to solve a particular problem. Or somebody might have done the heavy lifting already and packaged it up under a compatible license (if it isn't something simple like leftPad).

I mean, it's OK to ask for basic algorithms in interviews, to see if the interviewee has at least some understanding of basic stuff and/or can think on their feet and/or can reason about problems appropriately...

But I would still welcome it if instead of giving an interviewee a boring, memorizable problem like "invert a binary tree", the interviewer would find a more general problem and would see how the employee would tackle it, and if the interviewee refers to binary trees as part of the solution without writing out a full implementation from memory that's OK (as long as binary trees would be a valid approach to solve the problem, of course).

> the interviewer would find a more general problem and would see how the employee would tackle it

those are the best interviews imho. specially when both the candidate and the interviewer engage into a productive discussion about the solution the candidate gave.

at the same time, those take time and a lot of companies want a "fast" hiring pipeline, which leads to "invert a binary tree on this whiteboard!".

Just for reference, my company does not ask anyone to implement a binary tree or any well known algorithms in coding interviews.

I see a bunch of comments that look to me as strawman, or really bad interview experience.

A good coding interview should: - not be known by the interviewee if they spent even a few weeks on leetcode - avoid requiring anything that isn’t covered in an average CS101 class - draw its challenges from the need to process a new problem, and gradually uncover the edge cases in it and how to handle them gracefully.

I don’t do leetcode. But went now and tried the first hard question I found. It was “find the median of two sorted arrays”. I find this question to be a good example of very little background knowledge needed. (You’d need to know how to deal with arrays, and what’s a median - not everyone will know these, but it’s a pretty low bar)

This question already has plenty of room for mistakes, edge cases and problem solving. I’m going to take a wild guess from knowing talented veteran engineers I’ve seen that they can surely handle it in the interview structures I’ve experienced.

It’s not a question of status, but rather the utility of memorizing algorithms as an indicator of success. Notably, it’s probably not because Google exists and anyway the majority of one’s time isn’t finding the optimal algorithm.

I don't think the problem is senior folks believing answering questions about bubble sort beneath them (I'm sure some do but I doubt it's the norm), I think the problem is that 99.99% of developers at all levels don't need to know how bubble sort works to do their jobs well.

Guido is not a senior engineer.. he's a "special projects" person. Fewer than 50 such persons would exist in the microsoft stable. Why would you have someone who functions at such a high level bother with low level implementation? There's thousands of shit munching nerds who can implement flawlessly but who need the overarching guidance to deliver product.

Because most really good designers also did low level implementation? Thompson, Ritchie, Leroy, Torvalds. Heck, while I don't know for sure, probably Dave Cutler did, too.

I don't know any product that's pleasant to use that was developed in the way you describe. Python certainly wasn't.

One thing I have always hated about HN, is that they grey out downvoted comments. That has to be the absolute worst UI decision possible. At least at other sites, when you manually expand a downvoted comment, it's actually readable. Here the contrast is purposely so low, I'll never know what you wrote.

Many threads, SIMD, cache hierarchies, and calculations often being "free" due to general memory pressure means the classic version of any algorithm is pretty much guaranteed to be bad at it's job while a seemingly worse algorithm might actually perform better. You likely won't learn any of that from your algorithms book.

If you tried to implement those algorithms on my team without a very good reason, there would definitely be some serious conversations that followed.

In any case, vomiting algorithms on command doesn't seem to correlate strongly to programming ability. Almost all programming problems have a lot more to do with handling much higher-level complexity issues. The knowledge and art of knowing how to poke some undocumented third-party API to implement our system is much more practical. Knowing how to think at a high level is much more practical. Leave algorithm questions for research positions in that area.

Possibly a six hour panel interview on Javascript trivia, instead.

Wow, the replies to this actually saying Guido van Rossum should do an algorithm/DS leetcode interview.

It's so ridiculous to see these replies. So Microsoft should've sent Guido van Rossum a note saying, hey study algorithm/DS for at least a month and do 100 leetcode before you come talk to us or it's a waste of time, thanks, bye.

Discussions of the industry tech interview process are now poisoned by these factors:

* There is an entire industry built around tech interview prep now (books, websites, practice/mock interviews). Many would defend this practice because their paycheck directly depends on it.

* Many see this as a hazing ritual that protects their high compensation and often their egos as well. These people are often young and will eventually see how harmful these interviews are when they get older and need to switch jobs. But by then there will be a new generation of young engineers defending the practice.

> There is an entire industry built around tech interview prep now (books, websites, practice/mock interviews). Many would defend this practice because their paycheck directly depends on it.

Or you know... an algorithm and data structures class that's part of a serious Engineering/CS curriculum.

That's enough for passing familiarity with what the algorithms are / pointers to reference material when appropriate. Not nearly enough to perform on command from memory in 20-40 minutes.

I mean,

If someone knows he's interviewing at a place where there will be a coding interview, he would be crazy not to take a look at his algorithm textbook.

And maybe if the course only only gave the candidate a passing familiarity it wasn't thorough enough?

At the end of Princeton's Algorithms I-II on Coursera you've done fewer than 20 implementations, and those were open-book assignments with week-long deadlines.

A reasonable interview prep cycle would be closer to 200 practice problems, under time pressure.

>he would be crazy not to take a look at his algorithm textbook.

Why? what's wrong with going "purely" into interview?

an algorithm and data structures class that's part of a serious Engineering/CS curriculum

Good luck remembering the kind of details a typical interview asks for 10 years into the business.

Previously:

Google bureaucracy expected from Ken Thompson (!) to pass a C language exam (!!).

https://www.theregister.co.uk/2010/04/21/ken_thompson_take_o...

"So Mr Thompson, you say you have some programming skills"

I believe this is actually about checking in code, not about getting a job.

From what I understand it is like a "driving license" for each language - if you haven't passed the driving test you can still drive but you need to have an instructor keep an eye on you. If you haven't passed the coding test you need someone to review your code before submitting.

> I believe this is actually about checking in code, not about getting a job.

I'd say it's about doing the work for which he came there, from TFA:

"Google hired Thompson to create a new language, Go. But Google also requires all of its recruits to pass a language test. According to Thompson, he hasn't quite got round to it yet - and so can't submit code."

And no, I have no understanding for that utter stupidity, or any attempt to accept it.

As a Googler, that statement is just wrong. They were looking for a fun story and somehow got just the right quote to make one.

Code committed to the internal repo requires a review from someone certified familiar with that language's internal style guide.

He was hired to work on Go, not C, and just didn't get around to writing enough C to bother getting that certification. He can write C whenever he wants to, just like literally every other engineer, he would just get a style guide review at review time.

Indeed... I wonder why anyone would think ken would magically be familiar with Google’s internal C++ (not C!) style. I’m sure he would admit that he isn’t, and doesn’t deserve to be granted the C++ Readability certification just because he was involved in C’s development.

But it was not about some "certification." He states was not allowed to check in the C code he wrote, lacking a Google required exam. To quote that once again:

It's not and it wasn't about any C++ "certifications." He was involved in the creation of a new language. For the new language there could have not been any existing certification. And as far as I know, the initial implementation language of Go was not C++ but C. In creation of which was Thompson also directly involved. And as far as I know the people worked on the parts of the code before Google and these parts of said C code already existed before Google (Go had some dependencies on code from https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs and https://en.wikipedia.org/wiki/Inferno_(operating_system)).

So it was indeed as absurd as it could have been. He was a part of the team that directly invented both the language and the style guide of the language, but some bureaucratic exam was expected from him to submit the code to the source control system.

As I said above, the literal interpretation flat out wrong. There is no mechanism that exists now nor in the past to make the statement literally true as written.

I don't know how they managed to get that statement, but it's misleading. Many misconceptions begin with an element of truth. I'm guessing the element of truth here was the "certification" process. (Maybe it was something else, like a mixup having him as an advisor accidentally caused him to not be hired into the engineering group and thus there was a wait before he got engineering credentials to view/edit code. But that kind of thing is just silly paperwork and clearly fixed promptly.)

I just checked. He had literally written some C++ code before that snippet was published.

It was not the absurd statement it's been made out to be. It sounds unbelievable for a reason: It is unbelievable because it didn't happen.

But writing this you 1) just confirm that there are mechanisms by which he could have be not allowed to check in the code; 2) show how invested in the claim you are by comparing the date the Thompson statement was published and some date from the source control attributed to him, which again proves nothing -- his statement could have been true before it was published. I'm still puzzled by the passion behind "didn't happen" claim which still remains unsupported, while showing that the mechanisms for the Thompson's statement to be true unsurprisingly exist. I'm also surprised by your insistence on "C++" sources and "certification" there because neither is a part of his statement.

We are insistent because we have first hand knowledge of how Google’s code review and readability processes work.

I didn't get that impression, considering the uniqueness of Thompson's engagement: the major point from the start was that he was not an intern who came to add a few lines in existing C++ code. He was there with the bunch of old C (not C++) code which was already written according to different rules, and was to develop a new language for which he was to write the rules (with other such developers). Additionally, he was involved not only in developing that C code, but in initial development of very C language! Under all these circumstances, he apparently couldn't check in the code, for not passing some "test."

Your failure to even recognize all that (instead mentioning "C++" and "certification" which is not the same) and to address the fact that these circumstances completely differ from your own much more narrow experience with established practices when checking in C++ code for which the bureaucratic rules already existed simply doesn't tell me that you can be believed regarding organization's capability to not react bureaucratically. Which is what Thompson apparently said and I still find, as I explain, very believable.

That is how I searched in the responses for any claim which would directly support your opinion, and still haven't found it.

I have no problem to admit I was wrong, but I'd really like to see some direct proof. Like somebody investigating what actually happened there and getting some first hand account. Not the talk about "C++" and existing rules for that (for which there is up to now a confirmation of strong enforcement, not tolerance).

Ken is a friend and colleague. I worked with for years on the Go team, from the early days.

But ken was at Google a long time before the Go project started. (The Go project was not developed in Google's monorepo and was not subject to Google's readabiliy process, as is the case for many Google projects.) What ken was referring to was indeed the C++ Readability process (which doesn't stop anyone committing code, just requires a review from a person with the 'readability' certification before committing; as mentioned upthread). I know this from talking to him about it, specifically.

I can't offer you any more proof other than the fact that I was a witness. If you don't believe that, then I'm sorry? Not sure what else to tell you.

FWIW your conduct in this thread is extremely obnoxious. I'm sure you'll find a way to debate that, too, but please consider that I have no vested interest in telling you this other than hoping you might better learn how your conduct is perceived by others.

> I know this from talking to him about it, specifically.

Perfect! So what's the exact story, then, finally?

1) Was the quote from him completely invented?

2) Was he allowed to check in or not? Did he need the test to check in at that moment to which the quote refers? (I don't care about the moment when it was published).

3) And even more specifically: were there ever the times where he was in Google but when he not able to check in the code lacking some Google-prescribed "test"?

Just "yes", "no" or "I don't know" there, please. Also (just for my curiosity):

4) How long that period lasted?

Then you can elaborate where I was wrong by searching thorough what I've written -- I've specifically quoted the article and talked about his C background and involvement in a new language, never claimed anything about his C++ code and a "certification." I have an impression that your perception of what I've written and that, what I actually have written don't match.

> 1) just confirm that there are mechanisms by which he could have be not allowed to check in the code

If that's what you read out of the post, when I literally said the opposite, I can't reason with you.

You’re quoting a journalist paraphrasing ken. I was actually there. It was not as you say.

Wasn't Go...written in C?

It's mostly about getting used to the language usage style used in Google (the style guides are open, so you can see it).

The philosophy is that all Google code base should look like it was written by 1 person. It's great actually in practice: code in a big company should be hard to write and easy to read, as it's read by many people.

> It's mostly about getting used to the language usage style used in Google (the style guides are open, so you can see it).

> The philosophy is that all Google code base should look like it was written by 1 person.

But all google products look like they are written by only one person so the aim is achieved. The worst thing however is that other person started doing the same.

So Ken Thompson created go and gofmt so he wouldn't have to deal with that crap any more.

It is certainly not a driver's license, this makes as much sense as asking Hamilton or Vittel to do a driving test before giving them a job as a driver.

That said, I would not be surprised this would happen at Google as they truly believe their process is vastly superior, as they believe themselves.

Pretty sure they would ask Linus to to a C programming test as well, and score him badly because he can't remember the exact details of some algorithm he has not used since he wrote the first lines of Linux.

> this makes as much sense as asking Hamilton or Vittel to do a driving test before giving them a job as a driver

https://en.wikipedia.org/wiki/FIA_Super_Licence

As it said, it makes no sense at all. F1 is not Google, "Provided a driver has previously held a super licence they do not have to meet these requirements, a driver who has held a valid super licence for any of the previous three seasons is eligible for a new licence, "

>Thompson: I'm not allowed to check in code, no... I just haven't done it. I've so far found no need to.

He could probably get around this easily if he tried

Still a funny quip though

This is not true though. All code at Google goes through review. If you aren't certified in a language, your reviewer must be (or you need to get another reviewer with it). Unless Thompson's code is being reviewed by someone that has been at Google for just a few months, this is a non-issue.

This is equally interesting (or equally bad):

"Google: 90% of our engineers use the software you wrote (Homebrew), but you can’t invert a binary tree on a whiteboard so fuck off" [0]

More details two years later here [1].

[0]: https://twitter.com/mxcl/status/608682016205344768

[1]: https://www.quora.com/Whats-the-logic-behind-Google-rejectin...

No it's not, the creator of Homebrew has nowhere near the clout of Van Rossum or Thompson. I don't think it's a big enough or technically complex enough project that its author should expect to skip the standard interview procedure, whereas the creators of Python and C absolutely should.

to be fair; it sounds like he didn't get the job due lack of interpersonal skills; not for lack of programming skills.

It's like people forget the purposes of interviewing. "Can this candidate do the job we're asking them to do?". If the candidate has a proven, public, track record doing the things you're going to ask them to do, then you can skip the jumping through hoops part.

We use shitty algorithmic questions as a proxy for answering the above question (in the vast majority of cases).

That's assuming you want to hire the first person that can do the job.

The companies that popularized this were and still are EXTREMELY lucrative, and get a TON of competition.

They can afford to be choosey. So they can try to find not only the person who can do the job but also do it BEST.

And that's really hard to tell from just looking at their work experience. You want to see how their brain works under pressure, dealing with an ambiguous new problem they've never seen before.

A lot of people who rag on modern tech interviewing don't get this. Unfortunately, so do a lot of people DOING these modern tech interviews. "He said the algorithm was O(n log n) but the answer is actually O (k log n). Not inclined"

And so, we spiral.

> They can afford to be choosey. So they can try to find not only the person who can do the job but also do it BEST.

Selecting people based on how well they can code some linked-list traversal, in a web editor, while someone is timing them is not a good metric for overall "best". In fact you might reject candidates that think more about larger design problems, and overall your code might suffer.

>in a web editor

This is the part I dislike the most. They usually disable copy-pasting too because they know it's that easy to google so you can't just code in your IDE and paste.

Isn’t it usually both? My compy’s on-site has multiple coding sessions and multiple design sessions.

I've done my fair share of algorithmic hoop-jumping and have worked at a few of these companies over the last 10 years. I'm not convinced that the people performing these interviews are able to tell how a candidate's brain works at all in 45 minutes.

At some point it's all just signaling and wild guesses.

The solution to this is simple - any engineer applying for a job should simply say "no" and this practice will end. The real problem is that engineers in general do not know how to say "no", or they do not know how to negotiate. The role of college education is to weed out those who are not capable, so if you already have a degree of a reasonably reputable (not necessarily ivy league) college or university these type of interviews are completely unnecessary. Also, in the US the cost of firing someone is extremely low compared to europe so these type of interviews make no sense at all. In europe you would need to pay someone months of severance pay and what not, in the US you can simply tell them they are fired and that's it. The goal of these interviews is really to find something that can be used to negotiate your salary down, so don't be a sucker and don't fall for it.

Have you been an interviewer?

No. Why?

Wow, the replies to this actually saying Guido van Rossum should do an algorithm/DS leetcode interview.

It'd actually be really, really cool if more than few people of Guido/Linus/etc's calibre did go in for some of the whiteboarding interviews (under an anonymous resume of course).

As both of them explicitly acknowledge - they don't (or no longer) consider themselves as top notch "coders" as such. Linus has said something like "I'm not even a programmer anymore" recently. And Guido said something like "I'm not the best Python coder by far - on a scale of 1-10 I'd say I'm at most a 6". Not because they're lousy coders of course. Just that (in recent years) they've had far, far, far bigger fish to fry.

So they go in, and "fail" the contrived Sudoku / Knapsack / "build me an Instagram clone, please" problem... then get strung along and gaslit for a while, before receiving an email 6 weeks later tell them that they're "not a fit".

That would be lots, and lots of fun to see.

Not taking anything away from GVR, but I bet he does not pass leetcode medium/hards in time allotted without specifically practicing these for a while. I could be wrong.

I did a dozen or so interviews at MS. No one ever asked me if an interviewee "passed/failed". The only boolean was "hire/no hire", which took into account many more dimensions. But I know people that interviewed over 1,000 people, so my knowledge is more anecdotal.

I'd imagine no matter the problem he would breeze through an interview and leave the interviewer with some enlightenment on programming. It wasn't necessarily about whether the code was "correct". I remember once interviewing a ~20-year old for the summer program who when I gave him a problem he started talking about some linear algebra and geometry that I didn't even know were relevant but quickly made sense to me. But he was so humble about it, and looking to me like he wasn't sure if it was the right approach to take (It was hard to keep a poker face and not give away that he was a "definite hire" after about 5 minutes). His 5-liner to solve it had one bug but he clearly "passed" with flying colors. Unfortunately for us he ended up joining a startup in SV (and now is doing big things there).

With a known quantity like Guido, you just try to convince them how fun it would be work at Microsoft.

Notable programmers, those you know by name, are given positions that you get by having a conversation with a VP. Source: worked there for a long time.

Microsoft hiring is my centralized, and not all teams do leetcode interviews. Many non- famous developers have been hired there without leetcode as well.

"So, tell me about how you would develop a threaded application"

(Runs for cover)

> This is cool though. I'll be curious to see what he works on.

Well, after seeing what happened to Leslie Lamport, i don't have high expectations.

Miguel de Icaza: "The developer division at Microsoft now employs the language designers and contributors to Python, Java, JavaScript, Typescript, F# C#, C++.

We just need some PHP, Rust and Swift magic to complete the picture."

And Delphi, VB, erm J#.

Who is next? Ryan Dahl? Rob Pike? Larry Wall? Rich Hickey? DHH?

Larry Wall is at Craigslist

Seems appropriate

Craigslist is still web 1.0 no?

Anders Hejlsberg was the chief architect of Delphi and he has been with Microsoft. He also managed to create C# and Typescript.

Can we skip PHP?

No, what Microsoft should do is a WinRT language projection for PHP. Can you imagine the epicness?

Considering that php runtime in 2020 is quite speedy (see swoole) a WinRT binding (good lib space) is not the worst combination. Just the syntax :)

Facebook has that covered

No worse in 2020 than JavaScript..

Wait, who matches each?

Well, the TypeScript and .NET organizations live within the Developer Division, and now we have Guido so there's Python. We also have fairly substantial dev tooling teams for C++ and Java so I imagine some of the contributors are there. We also have Node.js contributors as well as Electron (due to VS Code). Source: I work in DevDiv as well.

Any ideas on Perl?

Nope, I don't know of anyone working on Perl stuff in DevDiv but it's a big division.

Microsoft has had a Python tooling team in DevDiv for almost 10 years now. Originally, we were primarily working on Python support in Visual Studio proper. We also took the first crack at Node.js support in VS - also before VSCode was a thing.

Aside from that, Microsoft also owns the VSCode extension for Go, so there's that team, as well.

Great idea to apply the principles of competition inside Microsoft:

After a few years of competition, they can just shut down the teams working with the least effective languages.

Using this methodology, Microsoft will end up using the best tools for the job. And they can accept people from any subculture for a few years while the experiment is going on!

This is the "developers developers develop" we have all been waiting for!

Simon Peyton Jones also work for Microsoft, so maybe add Haskell to the list.

No picture is complete without Urban Müller in it.

So they can own package management on Amiga OS, too?

Brainfuck. Mr. Müller is the father of Brainfuck.

Among certain circles, he was much more famous as the creator of Aminet.

And their contributions belong to Microsoft, whose corporate leadership then has tremendous power to influence the licensing and pricing of future languages.

My impression of Microsoft's plan for the forseeable future, on the developer side, is to capture value from open source projects with cloud hosting, GitHub enterprise billing and the like.

They want a golden path from their developer-facing tools (TypeScript, VS Code, C#) to their monthly services (GitHub, Azure, Teams). They're going to make all of that work together as well as they can so you won't want to look elsewhere.

Frankly, it's ingenious. If it works, the hard-to-monetize value of open-source projects like TypeScript and VS Code will be captured in your monthly Azure bill.

While we're at it, a reminder that the most popular extension for VSCode is Python support:

https://marketplace.visualstudio.com/search?target=VSCode&ca...

(full disclosure: I'm one of the people working on that)

Would it be accurate to say that after embracing open source with VS code they're extending it with the proprietary pylance?

I'm not sure what you mean by "extending open source". The classic EEE pattern always referred to specific outside standards or specs being "embraced and extended", like J++ did with J2SE. Pylance doesn't extend anything, nor is it a dependency that can lock anyone in.

I don’t know. I work for a fortune 100 company and the corporate IT department is purposefully choosing tools not under MicroSofts control. AWS, bitbucket, Slack, Intellij, etc. And developers are encouraged to use MacBook Pros instead of anything running Windows.

I do as well and it's the opposite. Their sales people seem to have their hooks into the decision makers.

If azure "performance crippled" Mysql is anything to go by they'll probably try to start extinguishing the open source components at some point - either by pushing people to use non open source equivalents or by turning their OSS projects proprietary.

I wouldn't be surprised if they've already roadmapped to end open source development on vscode in 2025-2030. The extensions are already proprietary.

In that list, TypeScript, C# were invented by Anders Hejlsberg who works for Microsoft. Python by Guido who also now works for Microsoft. F# was invented at Microsoft.

But as far as Java, JavaScript and C++, I am not who the luminaries are, that work at Microsoft. James Gosling and Stroustrup don't work for Microsoft.

They don't claim having inventors of the language, but language designers and contributors.

For example Brett Cannon is also at MSFT, he is a core developer and member of steering council for Python, he also qualifies along Guido for this claim in Python.

Herb Sutter, C++ ISO committee chair works for Microsoft, so while not the language inventor, someone pretty important in the community.

There's many folks from the Typescript team who sit on the TC39 committee that are responsible for moving state of Javascript (Ecmascript) forward.

Now that Microsoft Edge is powered by Chromium, there are many folks that also contribute/ have say in development of chromium and nodejs.

This is just a rumour, but Microsoft aggressively hired Chromium devs from Google.

Basically Microsoft wants to build the best tooling for the most popular languages. They have Github + VSCode. Who knows they may also acquire stackoverflow. It's in their realm.

> MS wants to build the best tooling for the most popular languages

If that is true, why don’t they buy out JetBrains?

Because VSCode is already doing better than JetBrains. It’s adoption as an IDE has been a rocket ship.

But I thought they wanted the best, not the most popular.

That might raise some antitrust concerns. Microsoft is already a strong contender the in the development tool space and there's not that much competition left.

Or maybe JetBrains is just not available for them. There can be something going on between JetBrains and Google considering Google decided to partner with them on the Android development tools.

Or maybe Microsoft feels they are doing well enough with their current tools. You can't really fit the JetBrains nicely to the "big picture". Microsoft needs VScode, because they want IDE running inside browser. They also need the big old Visual Studio. JetBrains tools could not replace anything, but they would be a third pillar.

lmao I assure you the department of Justice has bigger things to worry about than the IDE market

There was also that dude who created GW-BASIC.

He doesn't work for them anymore.

And the Pony guy

Mozilla recent layoff could be th prelude for that on Rust. And is not the guy behind Swift and LLVM - I forgot his name - hopping around anyway :). And Zeev and Andi from PHP are also on the loose last time I read something. So :)

Chris Lattner?

And Ruby!

As far as I’m concerned, they can continue to stay away from putting their typical Microsoft touches on Ruby.. NT ENTERPRISE, FOR BUSINESS.

Watching Microsoft from the 90s adapt and to see Satya get to do what he wants is pretty amazing!

I was thinking the same. I spent many of the formative years of my career (late-80's, most of the 90's) HATING microsoft but to see them now and all the relatively "open" stuff that they not only embrace but can't really extinguish is refreshing. I think some of that attitude put Satya in place, but he's run with the ball under the same ethos.

They only open that which they have no advantage in. Operating system is as closed (and as hostile) as ever, so is office and sqlserver. Xbox requires an always on connection.

It is their right, of course. But I don’t buy the “new Microsoft” ethos.

Win10 is still user hostile. A recent update on a computer I support for an older family member installed an uninstallable Edge, hijacked the PDF association, and with the first PDF opened through edge, used credentials stored in Office to log in to Microsoft.com “for convenience” without asking.

The monopolistic DNA still runs strong in Microsoft. They are behaving nicely only where they have no advantage.

I don’t buy the “new Microsoft”, not one bit.

Yeah, one side is doing great work - the Win10 story still sucks a bit / lot.

Completely agree. Just started a new job and was issued a new laptop with the latest MS stuff. Everything wants to be cloud based. The browser regularly has rendering glitches when scrolling a pdf. Little ads pop up for no apparent reason. Installed apps are just temporary unless you pay. It feels like a desktop designed by Facebook.

Having said that, aside from the pdf glitch I'd say it works very smoothly, exactly the way they intended.

I thought it was rather smooth too the first time I used it. Then I rebooted and the start menu was all messed up. Searched (from a web browser on a different system; not much documentation when you use it offline but at least no ads) and reason was... I changed the time. Somehow that reverted some of my changes to the start menu. Maybe a year or a bit more later, after I had arranged a few hundred program icons, the start menu just stopped working entirely and none of the suggested fixes I could find would help (except the one to completely reset the start menu and undo all the customization). But at least that led me to discover the super useful start right click menu (that still worked). Plus some strange audio issues with some programs that got much better in 2004 but still occasionally present. Plus the text entry box to search the app list to uninstall takes several seconds to even let you enter text (if you start earlier it seems random if it appears or not). And I'm fairly sure a few more such issues as well, so you are luckly if it is smooth for you.

But yeah, it does have a nice streamlined feel to it that I apprecitate (way better than the default Ubuntu interface IMO) except for the obviously hostile aspects. And the streamlined feel does really enhance the feel of how intentional many of those hostile aspects are.

I mean who's supposed to be the "good guy" then, Apple?

The FSF.

Why does there have to be a "good guy"? Can't there just be a bunch of different bad guys?

> Why does there have to be a "good guy"? Can't there just be a bunch of different bad guys?

Narrative tropes demand that there be a protagonist, so one of the bad guys must be cast as an anti-hero, at the very least.

Huh. I just realized that to many of his supporters, Trump is an anti-hero. No wonder pointing out his flaws doesn't work.

If Microsoft is supposed to be exceptionally evil then surely we can see other actors in similar position who aren't as bad.

IMO decentralization and competition are the only way to get the bad actors to occasionally do good.

Apple is the “least bad” guy at the moment. They are not “good”. But all the big guys are bad to some extent. (And since you brought them up - Apple’s cloud push is bad, though not remotely as bad as Microsoft’s cloud push)

I am not saying Microsoft is exceptionally bad in any way. I am just saying the “new, good Microsoft” is a mirage. They are as bad as they had ever been, except they have been beaten into submission in some sectors, so they behave in those.

Apple is the worst with their initiative to make their devices irrepairable, monopolistic practices on iOS and extracting value out of their target audience ruthlessly.

Apple does not have a monopoly - Android outsells them significantly in just about every market. Microsoft still has a PC monopoly. That makes a huge difference.

And yes, Apple making their device impossible to repair is evil, but their value extraction is opt-in (buy buying into their ecosystem) because they are not a monopoly.

What we have is an unregulated duopoly and Google and Apple are filling their boots.

The App Store is where they hold a monopoly.

I was referring to the legal concept of monopoly, of which they have non, and Microsoft does (and was in fact, convicted for abusing it, independently in the US and in the EU).

(The kind of monopoly Apple has on the appstore, much like the Sony monopoly on the playstation, and Microsoft on the Xbox, is not AFAIK an antitrust target because non of them command a significant part of a market without alternatives; whereas Windows on a PC was an antitrust target because it does have that hold on a market, AND microsoft abused it to enter other markets).

And Apple is the best (of those companies) when it comes to privacy and not spying on your every single digital move.

It is an open secret that Apple is doing all these privacy things for two reasons: PR and for setting themselves up as the only choice for ads for iOS users (which will soon include Mac). How anyone can just look past this (and that they jumped into NSAs arms and shared everything) baffels the mind. Apple is in no way less evil than Facebook or Google.

They're much less evil than Facebook in my book.

w.r.t google, it seems like Apple is the lesser evil - as far as I (and anyone I know) can tell, if you turn off location tracking, it actually stays off.

But there is definitely no "good guy" among the big player. Right now, Pine64 and Puri.sm are the only not-yet-proven-evil guys around, and they are both minuscule.

The evidence for MS being bad in the post I replied to is them forcing Edge on you. Well, can I uninstall Safari, or install any other browser on iOS that isn't a reskin of Safari? Not so far as I'm aware. The integration is much tighter than what MS offers.

I see it differently. The awful W10 stuff to me is a byproduct of adopting the FAANG PM crap - everything's an A/B test, everything's about driving engagement (even if they drive it with a nailgun into your skull :). It is a "new Microsoft", but the new Microsoft is parity with the existing Google and Facebook.

Can you elaborate on this FAANG PM stuff? I am not super familiar with it but it seems interesting.

The acronym is for Facebook Amazon Apple Netflix Google ... project management, maybe?

Product Managers and the common philosophies this title seems to have adopted at those sorts of places.

I was more wondering what about these philosophies are bad.

IMHO when you're strongly incentivized to optimize for engagement stats attributable to you, you're pushing change for the sake of change with the darkest of UI patterns to ensure your changes are selected and "favored".

FAANG-like organisations like to use statistics to drive decisions. As others have mentioned, things like "engagement", A/B testing, percent of users using a feature, etc...

This often leads to absurd decision making or false incentives. Things like the GUI shifting around making people click the wrong button by accident will increase engagement metrics and look like a positive thing in the reports.

Take Netflix as an example. If you set your UI language to English and live in Australia, Netflix allows you to choose only five languages:

- English for the hearing impaired

- Italian

- Simplified Chinese

- Traditional Chinese

- Vietnamese

Note that that list is for a show that has alternate audio languages like French and Italian, but you can't select all of those languages for subtitles. But here's the thing: Netflix has the subtitles in about 100 languages for every show. They just refuse to let you select them.

My parents are immigrants and like to listen to shows in English but have subtitles in their native language as a fallback. Netflix says "no". My partner is an immigrant and she's not Italian, Chinese, or Vietnamese. Netflix says her language just doesn't matter enough to make the list. I sometimes like subtitles, but I hate the hearing-impared subtitles. I don't matter either.

What's happened is that some overpaid statistician at Netflix figured out the top 5 most common languages for each region, decided that 99% coverage is "good enough", patted himself on the back, and the 1% can get fucked.

That statistician is probably paid 3-5x as much as me, and his job doesn't even need to exist. Just show every subtitle language available! Easy! No need to pay some "smarter than you" person to decide what you can and can't have.

Notes:

Apple TV shows every subtitle language, so there can't be any arguments saying that this is impossible or difficult or whatever.

If you call Netflix support, they will gaslight you and say that this is a copyright issue. It isn't, Netflix hides subtitles even for shows they produce themselves.

It's possible to do some combinations in any region. E.g.: setting the UI language to something other than English will make other subtitle languages show up. Some combinations are impossible though. So if there's some random couple like an Italian guy with a Filipino girlfriend, they can't do Italian language audio and Tagalog subtitles ever. Netflix has decided that their relationship is statistically unlikely and excluded them as outliers.

> They are behaving nicely only where they have no advantage.

This. Just look at gaming. Xbox is losing, so they champion pro-consumer cross-platform multiplayer. Minecraft is popular, so they kill GNU/Linux support with Bedrock Edition.

Bedrock Edition is primarily targetted at the consoles (where running the Java-based Java Edition was difficult). Java Edition is still worked on (and runs on Linux) and they are trying to achieve parity between the two versions.

Bedrock Edition was based on the C++ rewrite for Pocket Edition and has better performance and crossplay.

I think they're optimizing for a different use-case/perspective: > uninstallable Edge Alternative: The person has a browser that is auto updated and doesn't need to fiddle around

> hijacked the PDF association

Alternative: The person has an in-built PDF viewer without scouring the web for "PDF viewer" and downloading malware or a sketchy app

> used credentials stored in Office to log in to Microsoft.com “for convenience” without asking

Alternative: "Wow it's cool how all my Microsoft things are working well together"

I think there's room for grace about the experience of 99% of users who don't want infinite customization but something that works well with helpful defaults.

No, that person already had a PDF viewer installed, more capable than the one built in to edge.

There is no excuse to replacing that on a forced update without notification.

Seriously, you can explain anything this way: “all files deleted” alternative: “Microsoft just helping you free up space” or “the files weren’t backed up so they weren’t important”.

Microsoft is squarely in the “monopolistic bad guy” corner here.

Edit: Said person also had a self-updating Chrome, self-updating Firefox, and manually updating Safari on the same machine, for the record.

Changing existing defaults is def a no-no. I thought it was a fresh install of Windows you were complaining about.

> Win10 is still user hostile.

Maybe MS could catch the wave of "old-school" versions that are all the rage (e.g. WoW, Runescape), and release a Win95 flavour of Windows that runs on current hardware.

I miss the "old-school" GUI conventions and simplicity of - gulp - 25 years ago.

There are different divisions in Microsoft and these do different things. The division gave us F# (that I am eternally grateful for) is not the same that gave us Windows (that I am never going to forgive). These things can coexist.

You could say that about a democratic country. But Microsoft, like any corporation, is a dictatorship.

F# is public relations and research - so you get to enjoy the fruits. But the day it gains any strategic importance, it will be used and abused for that purpose.

With them sunsetting IE what's the alternative? Surely there should be a browser that cannot be uninstalled right? I agree with the rest of your issues, but if Edge is ever to completely replace IE it should remain installed no?

First, they haven't sunset IE yet. If the same update uninstalled IE, you might have had a point.

Second, putting the icon everywhere in your face is monopolistic behaviour. In Europe, for a while, Windows came without a browser installed, and let you pick one to download upon install. This is the non-monopolist way to make sure the user has a usable browser.

Also, that user already had a self-updating Firefox, self-updating Chrome, and manually-updating Safari on the same machine.

> Surely there should be a browser that cannot be uninstalled right?

No. You should be able to uninstall everything that isn't needed for the computer to boot.

The "new Microsoft" is useful for the time being. Once they are positioned to, they'll drop the hammer.

I find it frustrating because it looks like there are two sides of Microsoft. One helming Windows 10 and another doing the impressive Visual Studio Code, PowerShell Core, .NET Core work etc. where they go like "Hey maybe we should build a Rust projection for WinRT, OK here you go!"

Windows 10 should have been so much more and so much easier for Microsoft to rapidly evolve in a more beautiful fashion. But I guess there's just so much enforced backwards compatibility within it that their hands are almost tied behind their backs. Everything is ran in the same system on equal footing, from Windows XP era stuff to whatever the latest "app" trends are. Unsurprisingly it becomes a mess.

Windows 10X looked to be the right way forward where they finally decide to just run everything Win32 in light weight containers to shed all that dead weight for a more maintainable system, but it's sure taking a while to see anything and I hear the ARM edition that will be essential isn't going all too well. We're looking at a completely different, really troublesome, performance in 10X for x86 on ARM than macOS Big Sur and Rosetta 2 on Apple M1 that is already about to ship/shipping.

Honestly, I cut them some slack on the MS account integration (you log in to MS on one app and you're logged in everywhere) because Google and Apple both established it as good practice on their mobile OSes. In this case, MS was the last one to the party to make Windows integrate with their cloud accounts like that.

My biggest problems with Win10 are far more mundane - the incomplete/buggy touch interface, the explorer bugs that are old enough to drink, the new signed-executable security model that is very OSS-hostile (but again, imported from mobile OSes). The fact that half the configuration screens are in the new Settings system and half are in the old Control Panel, where you'll even bounce back and forth to tweak a single component.

I use Firefox because I'm not cutting that slack for Google/Chrome either (and I've never seen it happen in Safari, though I'll take your word for it that Apple does that sometimes too).

Not cutting them any slack.

Adopt, adapt, annihilate.

Yeah, I love the new Microsoft, especially because of the forced telemetry and updates. Not to mention making it impossible to create a local user account on Windows 10, unless you disconnect from the internet.

UWP is wonderful too.

It's not impossible to create a local account without disconnecting from the Internet.

Since Windows 10 1903 it's impossible, at least in the Home version.

https://www.howtogeek.com/442609/confirmed-windows-10-setup-...

Can confirm; recently configured a Surface tablet for family, and no workarounds were possible.

Microsoft should really improve their unattended process. This is where Linux really outshines Windows. I can configure a Linux installer with all my favorite software integrated, user accounts, configurations, etc—but with Windows I essentially have to install Windows, install my apps (leaving tons of randomly scattered files and registry entries all over the system), and then sysprep the WIM.

The unattended setup on Windows is very confusing, there are like 3 different ways and 6 steps to configure, and the tools are mostly designed with GUI in mind. Could you imagine being able to configure, install drivers and apps using a single YAML file for Windows?

I when’s through creating and deploying a Windows image last year. It’s a really painful and convoluted process and the tools don’t help much.

I think most of the more serious ones require an Active Directory setup, which kind of reflects their traditional focus on enterprise users.

Last time I tried (4 months ago) you still could create a local account while connected to the internet.

They're making it more obscure, though

Just did so successfully two weeks ago, Win10 Pro on T480s, Version 2004 or 20H2, can't remember. The latest of what the Media Creation Tool gave me, anyway. Others said Win10 Home does not work anymore.

Ironically, I installed Win10 coming from Manjaro, which even on a T480s (probably one of the laptops to have for Linux compatibility) broke after every other update, so like twice a day. On the other hand, Debian/Ubuntu has been too far behind on software versions for me. Both had terrible compatibility with my Thunderbolt 3 dock. Hopefully in the future, we will meet again!

Until then, it is ridiculous how much more Win10 Just Works.

I'm surprised you didn't try one of the more mainstream stable-but-current distros, like Fedora or openSUSE.

Perhaps they didn’t try due to the difference in package management?

Fedora/openSuSE are RPM-based distros while Ubuntu uses Debian packages.

For Win 10 Home, the only way is to create a temporary Microsoft-connected account to install, then create a local account after installing, then delete the first account.

Paradox of Microsoft's new image is that software development & tools has never been better as it is today, but the desktop experience is one of the worst in Windows history, the opposite of the old days. Seems that you can't have them both.

Yea I love this brave new world where frontend development feels like ASP.NET. Love me some TypeScript, shipping features quickly was getting old. Say it with me now, noImplicitAny! We're Engineers now, thanks Microsoft.

I've created local users several times. It's not super clear I agree, but definitely doable in the last month or so.

"Not to mention making it impossible to create a local user account on Windows 10, unless you disconnect from the internet." This is false. I've done this a couple of times recently on Windows 10 and it's no problem at all - just requires clicking a link on the create user account page. In fact, I always prefer to create a local account first and then tie my MS account to it afterwards.

> impossible to create a local user account on Windows 10, unless you disconnect from the internet

This isn't at all true. Accounts > Family & other users > Add someone else to this PC. Took me 20 seconds to find this setting.

Ironically they are still the same old Microsoft in most ways.

I would be curious to hear about other people who "retired" then went back to work.

I feel like if I reached the FIRE point, I would continue to work, but entirely on my own terms, not as an employee of somebody else. Even at my favorite jobs, where I got to spend the majority of my time doing interesting things, having to deal with red tape, institutional slowness, upper management, etc. was not fun, and I can't imagine it getting any better when you know you are now working only because it's your hobby.

At his level, there's very little red tape, institutional slowness and management to deal with.

As long as he doesn’t do anything that impacts the actual business. DEs working on Microsoft products absolutely have red tape and institutional slowness.

I have exactly that deal at MS (and I know many others who do), and I'm a couple levels away from DE.

I'm later in my career, don't need the job for $$ (they know it), lead a great team, work on a meaningful project that I pitched in an area where I'm an expert, great support from management. The only thing remotely close to "red tape and institutional slowness" is about making sure what we build meets the promises (privacy and compliance, for example) that we've made to customers.

Not only that but he probably has a small legion of full time developers working on his projects now that no normal FIRE savings would be able to afford.

I think it really depends. I could see continuing working for someone longer in an IC role if no one is breathing down my neck to do things I don't really want to do and, especially, if I could negotiate flexibility in terms of time off, etc. I'm actually pretty close on both those things today and, post-COVID, I could see making adjustments more in that general direction.

ADDED: As Denvercoder9 writes, some companies/groups can be pretty good about being hands-off a senior person who they want and also know doesn't need the job. Others, of course, are not. Certainly many Sun folks did not sit well with Oracle.

> I feel like if I reached the FIRE point, I would continue to work, but entirely on my own terms, not as an employee of somebody else.

I'm right there with you - this is my number one goal for FIRE, is basically to not have to put up with anyone else's idiosyncrasies but my own. The only way I could see that changing is if I see a group of people worth working with, or a project worthwhile enough to work with other people, or I would feel an obligation to be a mentor to others.

Careful with FIRE though. Don't fully sacrifice living with the goal of doing it later. I realize I'm not old by any stretch, but I'm at 40 next year. I've definitely made some sub-optimal decisions with money, but the experiences were mostly worth it. My retirement goal is to be able to do it by the time my kid is out of college - 14 years from now, unless she's smart and does the 5th year I never did. Anyhow, I digress, but I'm glad we traveled, ate, bought a house, sold a house, moved around a bit, etc. because even as income continues upward (along with some wealth if that's what you want to call our tech equity comp), life still gets harder in non-financial areas.

I'm convinced the worst thing we do to our kids is shove them through school and stick them behind a desk the day they're done. I didn't break into tech until my mid 30s, but for the folks who came out of college doing this, you have a legitimate shot to be set before you turn 40. That said, if I could rewind, I'd happily push my own retirement goals back a few more years in exchange for a financially secure year off in my 20's or early 30's.

What would you do in your financially secure year off in your 20's or 30's?

Not the parent, but I would have primarily traveled. Might have hiked the AT or something like that.

That said, although I didn't take extended time off, I did always max out my vacation time and took month-long trips to places like Nepal.

I bought a sailboat and lived on it while traveling up and down the east coast of the USA, and Bahamas. Lots of learning and character development.

I'd travel. I mean, we travel now, but not way off the beaten path. And we're mostly working around the school calendar (I'm ok being a little looser with that than my wife is). I'd also spend some time learning something that doesn't translate to money - get better at golf, work on my skating, etc. I'd also do enough nothing to be ready to go back to work by the end of it!

Triple Crown of Hiking.

Counterpoint: I sacrificed a lot of my 30s to become debt-free. 40 now and pretty close to FIRE (well I don't know about the retiring part, but the FI part is almost there).

It was totally worth it.

In the end, it's a personal decision. I do agree that if you find the majority of weeks to be acutely miserable, then you should honestly ask yourself if you're giving up too much present for an uncertain future.

I did as well, and we've remained debt free for a while. We're now in that cash poor phase since I seem to love pre-IPO companies :) Was it worth it? To some degree, but at the same time there are some big things on the list that we will probably have to defer for a while now that we have a school-aged child, a pet, etc. All I can say is, do some of those things, and don't fixate too badly on sacrificing everything to retire early. Family, friends, relationships matter, and so on.

Oh, trust me, I've definitely drank deeply of the pleasures of life and plan to continue down that road, just on a smaller scale. Seven course meal with wine pairings at Chez TJ in Mountain View when I did my Google on-site interview was definitely worth it.

I'm also in my early 40's. I've been working nearly 20 years, I'm just looking at another 10 years in this job, and I don't think I can take it. The being stuck behind a desk for a majority of my waking hours during my 20's and 30's definitely resonates, and despite my passion for software, I (relatively) recently re-learned just how good physical exertion feels.

So yeah, I cut back on a number of things to reduce my monthly burn rate, but I'm not suffering by any means. If anything, I'm happier now with less, and seriously considering if I need to continue working, or maybe I can coastFIRE. The possibility has not escaped me that maybe it's just not the right job for me, and perhaps I'd be better off consulting or even trying micropreneurship, but on a more relaxed schedule.

I do wish that when I was laid off in 2001 (my early 20's) I had hiked the PCT instead of trying to start a consulting business. Burned through my savings, and I probably would have just as much money at the end of the PCT through-hike, and definitely been happier.

We're in the same boat, but given that this is HN and I don't know if you're actually a billionaire or something, let's assume we both aren't. I'm not, for sure.

This is a rational position for folks like us who a) work to live, and b) probably haven't built anything to the level of the second most popular language in the world.

Now say you're Guido. I don't know him, but I feel like we're on a first name basis. Guido is clearly driven to invent, and has developed one of the major building blocks of modern tech. He's probably set for life financially, but Guido has the urge to innovate. Along comes Microsoft, who basically is paying him to do whatever he wants, however he wants. If you're Guido, you take that.

Now if you're me (and it sounds like probably you too), we don't have those kinds of offers, and I personally don't have that specific drive either. Ideal retirement for me is more time in my yard and also probably spending the cold months starting app projects that I'll never finish. The way life should be!

>I feel like if I reached the FIRE point, I would continue to work, but entirely on my own terms, not as an employee of somebody else.

If you want do to something that impacts or depends on other people then it will never be solely on your own terms. Plenty of people burn out of OSS because they think otherwise and then the angry users eat away at them. Likewise being FIRE doesn't mean you have unlimited resources so if you want more resource you will need to get them on someone else's terms.

I've known a number of senior people at these companies that basically get to do whatever they want, pitch projects and then have teams whose job is to work out the details. Their name provides so much political clout that the org bends around them rather than the other way around.

Personally I work at startups as a hands on manager so generally haven't had much red tape, institutional slowness, upper management, etc. hassles to deal with. Would continue doing so if I reached FIRE although might cut down my hours.

I think he is motivated by more than money and retirement.

Fun historical story about Microsoft and Python. I worked there for a bit back in the mid-90s when they acquired a startup who's merchant server software was written in Python 1.12 or something. After Microsoft acquired, it became the engine for running ecommerce on MSN, and eventually became part of the SiteServer suite. The first port to run the software at Microsoft involved wrapping the Python code with a shim that exposed it as COM interfaces. I think it was eventually rewritten in C++ with ATL or something, but that was after I was no longer there.

I still remember the first O'Reily pink Python book waiting on my desk back then when I joined.

Guido, IMHO the primary reason Python is great is because you have taste. Please insist on keeping that at MS...

Same for Clojure.

It's sad, dealing directly with extreme over-engineered enterprise Java libraries to compensate for Clojure's lack of ecosystem erodes all the attractiveness of the language for doing real-world or hobby applications.

OTOH, there is no much benefit today in Clojure over other languages for database backed apps, most of what is considered good Clojure code is more about the methodology than the language per se. The real benefits IMO is when you paired it with Datomic avoiding the impedance mismatch in data structures, but Datomic is another can of worms.

MS is soon going to show him who the boss is ...

MS is soon going to show him who the boss is ...

They let Anders Hejlsberg have free reign over his languages, encouraged it in fact.

Well, time will tell ...

Well, time will tell ...

He joined them in 1996, and has done J++, C# and now TypeScript. They seem quite content for him to just keep making new languages which they throw all their marketing muscle behind.

Someone on the internet: MS management will own him, like they do others, he hath been ingested into their belly

Me, some rando MS employee who works on languages: yeah, I can actually do pretty much anything that I feel is right, at half the level of Guido (if even that) and a miniscule amount of clout

Chiming in as another rando MS employee who works on languages, while I think "ms will own him" is thoroughly overly dramatic, I made this account to express honest surprise that you feel like you have leeway with little clout.

At least in my group our ability to do what we want was directly corresponding how connected/influential our sponsor was, and absent that, I have almost no ability to do what I feel is right, even if I have data backing it up, if the powers that be are in opposition.

I don't want to paint this as some universal ms-doing-terrible-things statement, just that I don't personally feel like I have much freedom to assert philosophy or ideology, and I've seen well respected individuals in engineering even up to and above principal/partner die on comparatively minor hills before which makes me think it's not just me.

And while I'm being dangerously candid, this to me seems like a prestige hire much like carmack at facebook, and I chuckle at the ado being given in both directions in this thread. I will be pleasantly surprised if this motivates more python support but expect neither good nor bad.

It will? Then make a concrete prediction - what should we expect to see to confirm your claim? What will “Microsoft showing GvR who’s the boss” look like?

Eh, I don't think so. If they ever do, he can just move on. I'm sure there's no shortage of big (or small) employers who would love to have him.

Yes, but who can pay him as much as MS can?

Google uses a lot of Python and I'm sure they'd be happy to have him, for instance

He already worked at Google from 2005 to 2012, although they might be interested in having him back.

Any of the other FAANGs or unicorns. I doubt that the paycheck is his primary motivation, but I could be wrong.

He worked at Google for seven years just after their IPO, and then Pre-IPO at Dropbox. It's unlikely that he's there for the paycheck.

Do people really consider Python to have taste? It basically ignored Scheme, ML, and Erlang dialects from its creation onwards.

I think it's all about the syntax and less about the semantics that still makes it a top 5 language.

Makes you think about how important the syntax for a language is.

The fact that Python code is readable like pseudocode is most definitely why I like it best, over esoteric languages like Perl or Erlang with their non-intuitive use of various characters.

To a seasoned programmer it may look obvious, but Python is still by far the best language for newcomers, and generally people tend to stick much longer to the language they were first taught.

I am "seasoned", and I find Python not obvious at all. It's a confusing language with little gotchas everywhere. I find writing code in something like Racket or F# to be much easier and more consistent than Python.

the user said the person who created Python had good taste, not that the language they created has it (somehow)

> the user said the person who created Python had good taste, not that the language they created has it (somehow)

I think that saying Guido has taste implies that Python is tasteful.

> Do people really consider Python to have taste?

Yes. Obviously people have different tastes, but many many people find Python to be tastefully designed.

> It basically ignored Scheme, ML, and Erlang dialects from its creation onwards.

Those are... all radically different languages from each other.

This is like arguing that a peanut butter and jelly sandwich would be better if it was more like sushi, chocolate, and pasta.

> Those are... all radically different languages from each other.

Not really, but that wasn't my point. My point was, that at the time of Python's creation, all those languages existed and have taste. The fact that they were ignored in Python's development is an example of it not having taste.

The best example of this is Python's scoping rules. When something like Scheme or ML exists and you end up with Python's scoping, it's hard for me to understand where the taste is.

I suspect Python's popularity was greatly helped by ignoring those languages. Your definition of taste isn't everyone's.

I didn’t say it was. I am curious though what others consider to be tasteful about Python.

It objectively has ignored features from those languages, such as immutability, concurrency, sane scoping, functional ways of thinking, pattern matching, better REPL, etc. which have all been making their way into most modern languages. So what’s tasteful about Python? The benefits it has that I see are that it’s easy to download and run for simple scripts and has clean-ish syntax, but that’s about it. For anything larger, it quickly gets in the way.

Popularity is a confusing beast. What has made Python popular, in my opinion, is that it was adopted by scientists, but in my experience, that group of people has little taste in software and little want to figure out what else is out there.

> The benefits it has that I see are that it’s easy to download and run for simple scripts and has clean-ish syntax, but that’s about it.

You say that like that doesn't count for a lot.

The ease of getting Python to do various small bits of automation is the main reason I use it.

It counts for a lot, but it’s not like that’s all people use Python for. There are also other languages with similar amounts of lesser friction to get going.

And those things aren’t enough to justify building large software out of.

Python's taste is one of ergonomics. Expressiveness + easyness to read, even for a beginner. Python evolved from a project that was testing the UX of syntax for programming languages.

My favorite example here is that in a function definition, `def foo(): \n pass`, the `:` isn't needed, a machine can parse it with only the newline. The : is a requirement because it made it easier for humans to read the language in user tests.

I just looked up some scheme code. "Tasteful" ? lol. Python's syntax is perfection.

Great, another person in the camp that hates parentheses at first sight. :) A language, and thus the taste of the language, is not just its syntax. It's the whole package. It just so happens that Scheme and Racket have a minimal syntax to enable more regular semantics and advanced macros.

I would recommend at least checking out Racket and giving it a chance if you've never even heard of Scheme. If you like Python's syntax, then you might be interested in the Rhombus project that the Racket team is embarking on or Pyret, both of which have Python-esque syntax. I'd also recommend F#, which has just as clean, if not cleaner syntax than Python.

It's hard to call Python's syntax perfection when it is missing piping and pattern matching. Python can't even do multiline method chaining without an extraneous "\" character or parentheses. Python also doesn't have a multiline comment syntax.

Even though I was proficient with Scheme, the parentheses were my least favorite part. I got tired of going crosseyed lining things up.

Doesn’t mean I dislike functional languages either. I just specifically found the parentheses more annoying than clarifying.

I can understand a dislike of parenthesis in the sense of it being a personal preference based on actual experience. I will admit that I was weary of them at first myself, but once I got into Lisps and particularly Schemes, I found myself liking the spartan nature of them. In a world full of complexity, it is nice to just get on with life, so to speak, when it comes to syntax, and I enjoy the regularity. F# is my favorite language, but I do hold Scheme dialects close.

ML is a strongly typed language, so it's hard to compare it directly.

Scheme is a better comparison, but its lexical scoping requires all local variables to be explicitly declared. You can argue that it's a feature, but that's rather subjective - and if somebody wants the simplicity of assignment-is-local-declaration, how exactly would you do it differently than Python?

FWIW, Python was originally written as a shell language for a distributed OS called Amoeba.

https://en.wikipedia.org/wiki/Amoeba_distributed_operating_s...

I think it would have seemed out-of-scope to include too many sophisticated features. Also GvR was young, he might not have had [enough] exposure to all those languages, eh?

Python 2 was pretty tasteful, simple, usable, the exception that proved the rule. I think Python has evolved since.

With Anders Hejlsberg also at Microsoft, maybe they do something like type-safe Python similar to TypeScript?

Anyway, congrats to both MS and Anders! This will be very interesting for Python.

> something like type-safe Python similar to TypeScript

That already exists, and van Rossum worked on it: http://mypy-lang.org/

Also Pyre [0] by Facebook and Pytype [1] by Google. More importantly, the core part (type annotations) allowing these libraries to exist has been added to core python for years now and keeps on evolving, so it is an ongoing effort.

EDIT: As mentioned below, also added Pyright [2] by Microsoft

[0] https://github.com/facebook/pyre-check

[1] https://github.com/google/pytype

[2] https://github.com/microsoft/pyright

You forgot the one made by Microsoft, Pyright (and PyLance).

Having tried a good number of these various type checkers, it seems like Pyright is the only one that properly supports all of the accepted PEP changes to typing.

Isn't that just F#?

In the sense that indentation has syntactical meaning, yes. Otherwise, no no no.

I don't understand. You can take Python code and basically write the equivalent in F#, ignoring many of F#'s additional features. It would look similar (probably even be cleaner) and would be type-safe.

F# is not even close of Python, they are very distinct

Of course, one is a sanely designed language. :)

Joking aside, my question centered around that if one wants a type-safe Python, why wouldn't you just want something similar to F#?

You can add typing to an existing codebase, saving you vast amounts of work compared to rewriting in a different language.

Can that really be done for Python and the vast amount of libraries? I don't get the sense that one could simply slap on a static type system on Python and get libraries typed for free.

That is unless we're talking about a gradual typing system for Python, where new Python code could be typed and integrated into untyped codebases, similar to Racket and Typed Racket.

The existing type system (see e.g. http://mypy-lang.org/) is gradual. You can mix typed and untyped code, at the cost of some safety.

You can also write type stubs for libraries that don't provide type information.

This is the same as asking for typed JS and getting Rust recommended just because the syntax is similar.

They are totally different, the core paradigm of programming is totally changed and most of the points of JS are not in Rust (meta-programming, garbage collection, community)

Agree: F# as a better Python

https://youtu.be/_QnbV6CAWXc

Thanks. :) I believe you're the only one who does here.

You always think that if one of these people joins your company it won't matter to you. You won't see them in the halls and they probably won't do "normal" work.

My wife used to work at Dropbox when Guido worked there. Apparently he was quite active on the company-wide mailing lists, even talking about mundane things like the food or parking.

So MSFT employees you may get a chance to chat with Guido!

Top of my wish list would be a python <-> VSCode experience at the typescript level of quality.

Interesting that mypy is a Dropbox (GvR’s previous gig) whereas Microsoft has a competitor python type checker, pyright [0]. Anyone have experience with it?

[0] https://github.com/microsoft/pyright

Why not use Pycharm?

Well, one good reason is that unlike VS Code, Pycharm is not open source.

How often have you ever needed the source code for your text editor?

The community edition is (Apache 2): https://github.com/JetBrains/intellij-community/tree/master/...

Thanks for sharing. This is really cool. And I'm slightly surprised that Pycharm uses the exact same codebase as IntelliJ, just with a different run configuration [0]. It makes perfect sense of course, but I was pleasantly surprised.

[0] `To run PyCharm Community Edition, please use the provided run configuration "PyCharm Community Edition"`

This is an unpopular opinion around here, and I use pycharm almost every day, but I think the future is vscode.

Reason is MS doesn’t need to make money on the tool, because it has a halo of services it can sell in and offer perfect integrations for.

In addition, MS has GitHub and thus privledged ability to learn from new public code as it is pushed.

AI-based code completion offered by kite and tabnine is totally being slept on right now, and MS has the resources to offer a very refined and constantly improving take on this massive dev services opportunity.

Jetbrains has said it has a long term project focused on this but as of now there is nothing.

I am not suggesting abandoning Pycharm any time soon but it is going to take a bigger company than jetbrains to keep pycharm in the race against vscode.

If the future is made up of tools so complicated that only giant corporations are able to provide them, I hope we'll also see a strong counter-movement of simple tools for independent developers.

I think the future is made up of powerful tools so simple that only great resources can build them. Simple tools that hide the magic are harder to build.

The fix is in because vscode itself will be forkable at any time. It will just be that _services_ from microsoft and other companies will be designed for the tool.

For example, this AI code completion behavior. I can say right now that if tabnine had a version working in Pycharm that did not break autoimports, I would already be paying a monthly subscription for it. Jetbrains should be offering something similar ASAP.

Another example would be PaaS for frameworks using Azure. Start a project, choose a domain name, instant staging and production deployments.

Anyone could build these kinds of plugins to sell in services. It will really be about who does it best. Though, I think Microsoft is specifically positioning itself for a long term play here and I think much of it centers around vscode.

I was thinking simple in the sense of less hidden magic.

Deployment can easily be done with a script from the terminal, which doesn't need special IDE integration. "Starting a project" doesn't need any tools at all in simple cases - one could try to keep cases simple.

Python usually has less need for auto-imports than Java does because APIs are simpler, names shorter and modules more coarse-grained.

You might not need an IDE and an AI with simple tools. Of course, some problems require or benefit from complicated magic tools, but perhaps not all problems do.

I see. Some folks may want simple, no magic stuff. My last product I had a one line terminal deploy for staging and one for production.

But setting that up securely in of itself was a lot of work.

On my new project I deploy by pushing to staging or master. These deploys are blocked if the commit under inspection does not pass tests.

That all happens because of docker containers and GitHub actions and secrets. There is an enormous amount of magic behind this stuff but I would not go back to provisioning and maintaining my own boxes.

I do think there is a lot of benefit to gain from having services built directly into the IDE.

For small projects, a text editor will do. Though linting and code formatting helps beginners and pros alike.

> MS has GitHub and thus privledged ability to learn from new public code as it is pushed.

It only takes a moderate amount of resources to collect this for yourself.

I meant this to be like last “firehose” access to Twitter, to get and incorporate new patterns as they develop.

But also to crawl and process the entire dataset including issues constantly.

It’s conceivable vscode could highlight and warn about known issues in packages as new code is developed.

It is hard to tell how Pyright and Mypy compare head to head. I've been using Pyright for quite a long time and have seen the huge leaps forward it has made. The experience feels a lot like how I remember PyCharm from a few years ago. I would really miss it if it were gone.

Microsoft is betting on pylance for VSCode instead of pyright, though, and pylance is not open-source [0].

I wonder if Guido will be ok with that. For me pylance looks like the biggest evidence that embrace-extend-extingish mindset is still alive in Microsoft, unfortunately.

[0] https://github.com/microsoft/pylance-release

> Microsoft is betting on pylance for VSCode instead of pyright

Not really "instead of", since pyright is the typing engine for pylance.

By far the biggest challenge for the Python ecosystem at the moment is how difficult it is for new users to get started, and how difficult it is for non-programmers to install tools written in Python.

I'd be thrilled if Guido and Microsoft could help make this better!

Seriously? python is the easiest language to get started imo, absolute minimum requirements to get a script off.

If non-programmers don't even want to run a few command in terminal like pip, why do they even want to start with python? The whole mentality is wrong.

I think maybe the bar is low for new to development experience across all fronts.

I coach newcomers through setup once every few weeks and it is a nightmare.

My aim is to get them to a state where they have a recent Python 3 on their path and they can create virtual environments for their projects which work in a predictable fashion.

Inevitably they have already tried several different ways of installing Python and their system looks like this XKCD: https://xkcd.com/1987/

Take a look at Thonny. It's specifically designed for new programmers learning to code, and it bundles Python, so it's as close to one-click as it gets.

https://thonny.org/

As for installing apps written in Python - that's really down to the developer. A desktop app would normally just bundle Python for its own local use, similar to what Java apps do, and users are none the wiser - to them, it's just another installer. Some examples of such software on Windows include Calibre and CHIRP.

"A desktop app would normally just bundle Python for its own local use"

I've been working with Python for nearly twenty years and I find the idea of bundling Python with my application daunting. I want it to be easy.

In this day and age, things like Google Colab, REPL.it, etc. makes it a breeze to start programming. You're basically two clicks away from coding, without any dependency hassles or other updates.

But I agree, it's a slog to set up everything on a computer, if you wish to install Python from scratch, along with some IDE.

Huh. A throwaway account here on HN said that he was working at Microsoft 8 days ago, and I assumed they were wrong and thinking of Dropbox: https://news.ycombinator.com/item?id=24983394

Sorry, throwaway account! Apparently I was the one who was wrong!

GUID-o. Hehe. I'll show myself out.

Here comes P#, I'm seeing myself out as well.

Hey, that's what I tweeted to him!

> Here comes P#, I'm seeing myself out as well.

I tried to convince Guido to do a P# April Fool's (I think in 2005 or 2006), but he disliked the idea so much I didn't do it myself either.

You joke, but IronPython was a thing.

P# was already a project at microsoft. https://www.microsoft.com/en-us/research/project/safe-asynch...

It has transformed since into: https://microsoft.github.io/coyote/

Python for Excel data manipulation would be great.

Office suite automation with Python would be great across the board. The external python libraries that do this are pretty clunky and VB scripting is a hot mess.

Office now allows you to script it using Typescript, so you don't have to use VB any more.

Oh that’s cool! Haven’t had to do anything with it in a few years, but excited to check out how this works.

I'm really curious what you are talking about?

I could only find "Office Scripts", which doesn't work in the normal desktop apps (only the web version).

Edit: I just found out about "Script Lab". Is that what you are talking about? https://appsource.microsoft.com/en-us/product/office/WA10438...

https://www.xlwings.org/ and https://www.pyxll.com/

Agreed. VBA makes by fingers and eyes ooze gangrenous pus.

First-class integration in Excel would be a game changer. Holy shit. Though adding a multiline editor with indentation in Excel would be really nice as well. We've got to stop pretending Excel functions don't constitute a programming language.

Bring in versioning/snapshotting with some git integration while you're at it...

The finance folks I work with are fucking wizards. I'm convinced many of them would have made great engineers given what they can accomplish building applications in what might be the world's worst IDE.

Well... most quants are recruited straight out of engineering schools so you are probably right. About the IDE, honestly once you think about it, it actually achieve many thing at once. I am not sure what they could change to make it better.

Some of the most impressive, most complex Excel apps I've seen were 1000% not built from quants out of engineering schools. That's the beauty of Excel - but _managing_ these apps is the nightmare of my specific profession :)

I'd say two things would make it better: - Have a "code" version of the cell editor that has formatting/indentation + basic linting. [Sometimes 75% of the challenge is reading the code in linear fashion and figuring out what it does]

- Integrate git-like versioning and some sharing features with Office365. Basically, make it so that you can roll-forward or roll-back on both the code and the contents, and make it so that you are _not_ emailing files around to share them.

Interestingly enough, Libre Office (and OpenOffice before that) allow you to use Python internally for Macros through PyUNO, which is a pain to use given the lack of documentation, but amazingly powerful.

It was one of my first usages of Python, and used that to create quite complex Spreadsheets

It's been at the top of their public feedback request page for ages: https://excel.uservoice.com/forums/304921-excel-for-windows-...

They did officially respond in 2018 and put out a survey for thoughts, don't think they've followed up since but they definitely know demand is there.

Being retired is more boring than working for a giant bureaucratic corporation?

He must have no imagination. When I had a day job at a big company, I would start working on open source in the afternoon as soon as I got home from work then I would stay up late working on it. The next day I would be tired and depressed that I would have to go to the office to work on some useless product and could not wait to go back home to continue the real work.

Imagine if your day job at a big company (with the attendant salary and benefits) were to write open source?

Speaking of Python in particular, there are people at Microsoft who get paid for, among other things, working on Python itself.

Retirement has always sounded boring anyways. Congrats to Guido, I wonder what he will be working on specifically. Python could be improved on Windows, even simple cases don't work quite as expected always. I would expect Python is used decently widely at MS anyways. Maybe Python being first class on .NET? Or some Python specific things for Azure.

There already is IronPython. But it lacks community, and stalled after 2.7 release.

All CPython alternatives stall do due to the widely used C API (required for NumPy, TensorFlow, etc). This API is very CPython specific.

There is some work on both sides, that might overcome the issue though: CPython is trying to decouple API from the implementation details, while .NET is working on better C interop story (.NET 5 just introduced ability to export managed functions as C functions).

Congratulations to Microsoft on successfully hiring a developer with 30 years of Python experience. Living the dream right there.

He should make another language, surely that's a good retirement project? But he must call it "monte".

Well Anders Heilsberg did exactly that from Pascal to C# (we gracefully ignore delphi and j++ in between). And the after C# he thought about Types in JavaScript. So, I do not know Guido, but with language designers you never know.

I’d really like to see move emphasis on types. Working in C# lately made me realize once again how shit it is writing non-trivial applications without strong type checking.

Am I the only person here that finds this somewhat terrifying?

The internet has a lot of people - you're probably not the only one scared.

Care to explain what's worrying you though?

EEE, github, windows 10, there's a lot of reasons to suspect that when Microsoft becomes directly involved in something it starts to become less useful to the user.

They hired Guido not the PSF.

See: https://www.python.org/psf/sponsorship/sponsors/

Hopefully Microsoft will make Python a first class language internally. There's still far too much pressure to use C# for everything.

Anyone remember Boo? [1] It was a Python language that ran on the CLR/DLR VM of .NET. It was used early on in Unity when it had C#, Javascript and Boo. It was the language that coroutines was first implemented in that used Python like generators behind it.

There was also IronPython added when the DLR came out and is still updated to this day, latest works in .NET Core. [2]

It would be fun for Python to have more focus with Microsoft tools and some things on Azure or even another Boo/IronPython like CLR/DLR virtual machine language or more focus on IronPython itself as well as more of CPython. Having Guido at Microsoft is awesome.

[1] https://en.wikipedia.org/wiki/Boo_(programming_language)

[2] https://en.wikipedia.org/wiki/IronPython

The problem with IronPython, and really most other alternative implementations of Python, is their inability to reuse native code packages, because CPython extension API is so low-level and tied to the VM's implementation details (and that cannot be easily changed now for backwards compatibility reasons). Such packages are fairly common, and if you take them all out, it significantly reduces the value of Python as an ecosystem.

If you're using IronPython then you're probably just trying to embed a familiar scripting language into something, not trying to adopt the Python ecosystem.

Besides, while you lose out on Python's native ecosystem you gain access to all of .net's.

I'm not disputing that IronPython can be useful (although one also has to consider https://github.com/pythonnet/pythonnet, which wraps CPython into an IronPython-like projection). But if it can't share the library ecosystem, it might as well be a different language for all practical purposes. And it turned out that there aren't enough people who were interested in a Python/.NET combo for its own sake, so it remained niche. Any new implementation of Python on top of .NET VM would likely get similar reception for all the same reasons.

Why do I get a bad feeling about Microsoft’s open source effort? And it is just getting stronger by the years.

If it is any indication, I have seen a spike in activity in IronPython3 repo [0], which previously looked abandoned.

[0] https://github.com/IronLanguages/ironpython3

Yay, that's exciting. IronPython is necessary for scripting into my .Net/C#-based Revit but it's been stuck on IP2 forever.

Anyone using Pylance? It's a new Python language server implementation from Microsoft, and it's great. With 3.8's type hinting, you basically get a fully typed language, similar to what TypeScript did for JavaScript.

I’m excited to see what kind of teams they get going that focus on Python tools!

Start with rewriting the Azure Python SDK.

Honest response, if somewhat off-topic, since I can't pass this up: How could it be improved/what's been most painful for you?

Full disclosure, I'm one of the <many> maintainers working on a subset of the Azure Python SDK. Currently, there actually _is_ a large-scale rewrite in progress to bring the various SDKs up to a consistent level of quality and python standards, since it's no secret the original batch of SDKs grew rather organically. As such, this is a VERY APT time to hear this sort of comment. (And yes, I'm taking it totally deadpan even if it wasn't necessarily meant that way :P)

Do feel encouraged to file issues on the azure-sdk-for-python github as well as things come to mind; there are more formal triage processes there than "I happened to read this over lunch" :)

Hi, didn't expect such an offer, and the original comment is made in a more ranty tone than it should have, but let's try.

Disclaimer: some (or maybe all?) of the things I'm mentioning here might be caused by the API itself, rather than the SDK. The order is in which it comes to mind, rather than importance.

1. Some kind of strongly typed errors. I've turned the SDK upside down, and the only error that seems to pop up 90% of the time is CloudError with CloudError data. So our SaaS application (which utilizes Azure IaaS heavily) has a bunch of regexes that parse the messages contained in CloudError and throw something from our domain. Wrapping SDK errors by the implementing clients is a smart thing to do, but not this way :). Some messages change from time to time, so we try to make our regexes more tolerant, but you get the picture. In general the error handling is very strange, a lot of low-level errors sometimes flowing into high-level calls.

2. Documentation with more examples. The easiest thing to do is sometimes looking at the API or CLI documentation, and try replicating it with the SDK. Parameters named differently etc.

3. For some reason the Storage API/SDK keeps changing and it's like it's from another world than compute and network, which look pretty similar. The storage portion is... wild.

4. Improved working with data disks. Updating the VM with an array of disks has shown bad performance (and Azure support helped us and said that it can't be improved because it's a cross-RM event) but also poor reliability. A separate set of methods to attach/detach disks to VMs would be great.

5. Because we heavily rely on asynchronous operations, what we usually do is make a raw HTTP request for an operation (like VM deployment), serialize it and store it, then after some time deserialize it and check if the operation is done. We do the same with hundreds of requests at the same time, being able to process a lot of cloud operations with not much resources on our side :) . It works, but I found the raw request and serialization is a bit clunky. We're also stuck with a couple of older SDK libraries because we've noticed the raw responses changed in the newer ones, so now we need efforts to find all of the differences and consolidate the serialization. For example, the headers have had multiple urls to check the deployment status, sometimes it's `async-url`, sometimes `location-url` etc. This probably isn't something the SDK is to blame, and this is a niche case, but serializable async responses would be nice :D .

I have to commend the effort to break up the libraries from the original SDK into independent units. The versioning and dependency management became easier than with than one monolithic "4.0.0" schema.

Maybe some of the things are out of scope and our use-case is disproportionately affected by some stuff, it's just my two cents. I'm less satisfied with Azure as a service, with its unreliability, sub-par support, idiosyncrasies and poor "hypervisor" performance, than the SDK, but it's the SDK that gets all the blame because it's the one we're interfacing with.

Hah, feel no regret about the potentially ranty tone, frankly, I often prefer to get feedback from people who are willing/able to rant, since they're both willing to be candid AND clearly have some interesting thoughts built up.

I'm honestly appreciative that you'd give this long a response, I've unironically been sharing this with peers internally for what an interesting spread of feedback it gives. I have to give a similar disclaimer of "no promises" BUT...

1. 100% agreed with you. We're currently working through this discussion for ServiceBus, actually, it's VERY apt you mention this now, as we've been working with the architects and other-language SDK owners to find proper semantic unification between Python's inclination for more fine-grained strongly typed errors, vs. e.g. dotnet's inclination to have unified exception types and "code"/"reason" fields. Can only speak for the data plane SDKs here (vs. -mgmt, that's a different team) but our architects have within the last 2 days expressed their alignment with the pattern you're requesting. (We explicitly as a guideline want users to not have to rely on any ad-hoc string parsing to distinguish exceptions) We've also been attempting (to lead into #2) produce both better samples EXPLICITLY to show common failure modes and "best practice defensive coding", but also conceptual guidelines (e.g. here [0]).

2. Preaching to the choir. Samples are a First Class Entity for our Track2 APIs (complete with validation, smoke tests, etc), used not only as inlined samples in docstrings and refdocs, but as long-form examples of E2E scenarios as you suggest. (and for more esoteric subject matter that may not be covered in primary long-form docs)

3. I'm not 100% sure I'm thinking of the right component when you say Storage, do you mean e.g. `azure-storage-blob`? #4 implies you may mean more of the block storage/VM interconnect logic, and unfortunately I can't speak with deep familiarity surrounding that, other than to note that the ARM template parameters for configuring this hasn't always been "great" (e.g. I think I recall there being a need to format a disk as a separate step? not sure if this is the sort of thing you're referring to, half thinking out loud, but that's more distant from my area) Would be curious to know specifically which transition was painful though/what kept changing, since it is a "core awareness" going forward that migration pain is a Major Friction Point for users ("well duh" says every dev ever) so I'm just curious to see if your specific callout is something we'd have been aware of/can impact.

4. See the above re core surface area of "working with disk"; and to your later point some of this may be more service-side than SDK, but your mention about attach/detach disks is salient and something I can perhaps throw a shout at someone about.

5. To make sure I've understood, you're serializing like, metadata/a record of the long-running operation and doing it yourself? The track 2 APIs use something called a Long Running Operation poller internally to facilitate this sort of operation; now this may not solve your scenario depending on how your control flow gets passed off or not/your need for serialization, so this may not actually change anything tractably for you, but mentioning on the side in case. Curious re: what you mean by "is a bit clunky" though more precisely as well. And in terms of raw response consistency, yeah, that's not surprising, that's not one of the things I think we pin in backcompat (or even feasibly could, to your point, BUT we may be able to take more into account if there's any way to offer a continuous experience if this changes and folks were relying on it. Regardless it's something for me to keep in mind when doing design discussions with the service teams)

Glad the library fragmentation is well received. I'll candidly admit I was worried about that (was a user, not a maintainer when it happened, and did the normal grumpy-engineer thing of "they moved my cheese") but it does seem to have worked out nicely and given some good modularity.

Finally; Two cents, this was ten cents! I feel like I owe you big time for having given such a well-thought-out response. Thank you; sincerely, you didn't have to do this and have done us a big favor.

Absolutely get where you're coming from re: SDK being the public face, and that's what leads me to be eager to try and improve what we can. (And selfishly gotta make sure I can feel proud of the code that has my name attached to it :P ) I cannot tell you how much Reliability resonates with me. If nothing else, I can assure you this dev is in your corner; hardening, reliability, stability are the mantras within my purview. (signed, an-artist-formerly-known-as-SDET.)

[0] https://github.com/Azure/azure-sdk-for-python/tree/master/sd...

Hi, nice conversation going here :) . Let me just reiterate on a few points.

3. Correct, I'm referring to azure-storage-blob and azure-mgmt-storage. I'm not concerned with the unmanaged disks so much, which utilize the storage accounts (thank god, we're over that :D ), more with working with regular blobs. From the initialization of the BlobClient to the operations which are sometimes weird in their input parameters, it's not up to par with lets say azure-mgmt-compute.

4. This point is not related to point 3. I'm completely referring to managed data disks, which are first class* entity (like NIC, for example, not a Page Blob any more). Attaching them to an existing VM is a nightmare, and because of some features we do that often. Also they lack basic things like per-disk disaster recovery (but this is way out of the scope of the SDK).

5. Correct, we're doing it by ourself. The LROPoller is also there in other parts of the SDK, we also use (if it's the same poller), but I think we were not able to utilize it. Ny clunky, I meant the response itself sometimes has headers with redundant values, we're not sure which one to use so we just take from experience (if there are two similar) etc. I haven't looked at the docs recently, but the "raw" requests are not documented that well.

* - I did say that managed disks are a first class entity, but we found out the hard way that the managed disks and managed snapshots are actually just in some storage accounts on the backplane, and when something goes South, storage-level errors leak out to our logic :) . Also throttling, thresholds and quotas apply as if it were just another storage account, but even worse, we don't control how the disks are distributed between those backplane storage accounts, so we get funky issues from time to time. We solve them by being extra careful and conservative with some operations, but heh, talking about leaky abstractions. This is not an SDK thing, just wanted to point that out for someone reading.

Yeah, agreed that the raw requests were/are somewhat clunky/not well documented (they were added at one point as an escape hatch for when things didn't work as expected/the service did unexpected things for some requests). I would love to hear about what weird input parameters you are seeing in the azure-storage-blob package, however.

And +100 on not having to manage quotas/storage accounts/figure out how many accounts I need to distribute disks across in order to avoid throttling. There are many things I'd rather do :).

Man, there's so much stuff I want to do but don't have time for. I can't imagine retirement being so dull that I would go back to corporate work.

I really dislike Windows but Microsoft is doing everything it can lately to please developers. It's kind of working, I have to say.

Good news for you: .NET is really, truly free of Windows now (minus a UI answer).

I know. I'm more happy with the Linux subsystem, a proper terminal, etc. That kind of move.

Why do I expect a type safe Python? I'm sure the product I'm imagining will also finally give a Python inspired/compatible language a decent module system, but I'm really not a big fan of the direction Microsoft is carving out for software development at large. I'll admit this post is presumptuous.

I wonder about such successful but dedicated to a single project persons - don't they have a FOMO?

Like Mark Zuckerberg?

And Bob Dylan?

Van Rossum is famous for the Python language and language-related stuff for Python, but he did stuff in his career before inventing it and has done stuff since (for instance, a lot of his work at Google was on App Engine).

FOMO on what? Python makes more of an impact on the world than most people would even dream of for their projects. The products that use the language cover the entire spectrum of digital technology...

Yes, it is a very popular glue scripting language (to oversimplify a bit :)) but that's not the point, I mean that Guido himself is not involved in all of that spectrum directly (AFAIK).

I'd say it's really exciting to stay focused. Plus Python has been the hot baby for quite a while so he is not missing out anything IMO.

Classical Google lineage: "From 2005 to December 2012, he worked at Google, where he spent half of his time developing the Python language." Now he'll spend the rest of his life at Microsoft where he'll giveaway the Google-investment.

And will never make as much money as Google or Microsoft out of his own work...

If this includes improving dependency and runtime version management, I'm pumped. It's been a couple of years since I've used python, but those two things seemed like such warts compared to the rest of the experience.

That's all well and good, but it drives me crazy Microsoft refused and refuses to invest in F#, which could have very well been the Python of today if they had paid any attention to it.

> F#, which could have very well been the Python of today

There's a phrase you don't see every day.

As another F# enthusiast, I share the sentiment, but practically speaking I don't think F# would have been in any position to be the Python of today. It might have had a greater chance if .Net Core existed 15 years ago, but being tied to Windows was a big show-stopper regardless.

I agree that my comment takes into account too much hindsight, but certainly this is part of the pain. If Microsoft had transitioned .NET to be cross-platform much sooner or from the beginning, then they would have been better off. I was recently surprised to learn that Mono basically existed from the beginning of .NET, as I had no idea it was so old, so it's clear that people wanted something like .NET Core from the beginning.

It's confusing to me that Microsoft embarked on their Iron <x> project, or whatever it was called, to stress test the CLR, got F# out of the project, and then decided to sit on it. They have incorporated F# features into C#, but it is confusing to me that they saw the rising popularity of Python and decided to let it happen unchecked because C# will never be F# or a language that appeals to Python programmers.

It worked on Mono

> which could have very well been the Python of today if they had paid any attention to it.

Wow, really? I'm a big fan of Clojure and F#, but IDK, I think there is a bias against functional languages since they, IMO, are harder to start doing stuff.

You can write F# just as you would with Python. This would be just using OOP or just a collection of functions operating on simple struct- or dictionary-like types, using for loops, using list comprehensions, using if-else, etc. It will look very similar and be just as easy or easier to understand.

F# is strongly typed as I understand it. Python succeeded because it wasn't. A languages type system design mutates slower than the domain specific needs of what is popular. That slowness translates to difficulty in grabbing emerging domains. And the existing users are opposed to untyped structures as hacks causing friction. For example, most deep learning libraries even in typed languages are only semi-typed as matrix sizes aren't compared in the type system but in run time.

F# is a community with an independent mindset. That is not a good investment target. Also there is a factor is 100:1 between C# and F# developers. I think the investment is maybe even in favor of F# compared to that.

Ironically I had just stumbled across his LinkedIn profile for some reason last night and it was "retired, recruiters please get the memo" or something of that sort.

I don't know about anyone else, but I'm still hopeful for Python integration with Excel. That would be a real gamechanger, depending on how tight the integration is.

http://hugunin.net/microsoft_farewell.html

I'm quietly hoping they'll ask him to work on a typescript-style of project that adds static typing as a layer on top of Python :)

+1 for easy compiled binary in single file for all platforms. Bonus: Ensure support for CEF / PyWebView / WebGL on all platforms with secure IPC style bridge to the Python logic. Ultimate Combo: Smooth animated visual UI with secure connection to fun intuitive Python data services layer, all in one convenient compiled binary. ( Double bonus: Cython style transpilation step to protect source from tampering )

Is this a sign of the path for Python to finally become part of a typical web and consumer toolchain?

> Is this a sign of the path for Python to finally become part of a typical web and consumer toolchain?

Maybe? A likely first step would probably be something like an Electron fork/distro with Python built in, along the lines of the abandoned PythonWebkit: https://www.gnu.org/software/pythonwebkit/

Maybe Python will get nice tooling.

I wonder how working at Microsoft will turn out different than working at Google did?

They really try to own everything Python related. I fear in 10 years the only viable way of using Python is on a Windows PC with Visual Studio or some other Microsoft owned IDE. You have to be logged in with your Microsoft account and everything you type will be processed and sent to a Microsoft data center.

Yeah, that's a valid fear since that's how it works today with C#, TypeScript, etc.

Is he being hired as a Distinguished Fellow on par with Miguel de Icaza or Anders?

Distinguished Engineer.

Do you think Microsoft can convince him to drop the whitespace-as-syntax?

Why? That's my most favorite part of Python. It makes the code so much more readable.

Ruby manages to be much more readable without the white spaces.

The fault with Ruby is its less useful stb lib and lesser performances.

IMO Ruby is far, far less readable. Way more magic, endless ways to do the same things with slight differences (blocks, procs and lambdas) etc.

Plus everyone mostly indents the code the same as Python anyway... so. Meh.

In the age of autoformatters it makes moving code around more painful.

In C++ I can write some janky looking code and autoformat it into respectability. I can stop caring about whitespace entirely (while still producing readable results).

In python I need to write correctly-indented code because the autoformatter can't figure out if the lack of indentation is meaningful or not.

That seems like a silly argument. That would be like saying I don't like languages that require semicolons because then I have to remember to put in semicolons.

The whitespace is part of the syntax. Just like in some languages putting a semicolon at the end is part of the syntax.

And for the most part, your IDE will take care of whitespace anyway.

clicked the link, got:

"Something went wrong, but don’t fret — it’s not your fault."

Took me a moment to realize that statement wasn't in reference to the statement "Guido van Rossum joins Microsoft".

Hopefully he is going to make a really good port of Python to .Net.

Does this elevate the future possibilities of Pyright in any way?

Update IronPython!!!

VBA out PYTHON in

hey hey hey...

MS has been very timidly introducing JS as a replacement.

I hope that when Microsoft announced Windows 10 is the last one, it meant that the next will be Linux distribution able to run Windows apps better than Wine.

Typescript for python?

Good for you Guido!

python will end up like FORTRAN or COBOL

From now on known as Darth Rossum.

Any guess on what a major FOSS dev grosses? It has to be at least $700k. $1.5m/yr? $3m?

About $200k/yr+, it's a good idea to find a lawyer or similar talent agent to workout the compensation package because they have the skills and can get away with things a hire cannot. I've done this before, it saved time, and ended up with a better package.

levels.fyi gives $650k in SF. But it looks quite low to me. I'd have say 1M/year at least. Any better guess?

I think you're right. The issue is, due to the power law distribution of talent, the sample size of such roles (and personalities, i.e., how many Python BDFL's are there?) is incredibly small, so it makes precise assessments difficult and the variance increases wildly since there aren't very good comparables.

I think he could command nearly $10 megabucks, give half of it away to charities, grants, scholarships, social ventures, and underserved areas to promote STEM college prep, and still have more money that one ostensibly average, middle-class person would know how to invest (real estate on the near periphery of burgeoning urban sprawl areas). Microsoft is a cash cow and is willing to invest in a solid, deep bench to stay competitive, especially in an apparent transition to either EEE or adopt OSS.

Someone else commented that he'll be Distinguished Engineer (DE). But levels.fyi shows 69, the level under DE, some reports of 1-2m. He's almost assuredly receiving 1-3m compensation.

The ones that let other people do the work or the people who do the work themselves? A lot of prominent folks in the Python space are the former.

Sources familiar with their thinking say Torvalds and Stallman will join Microsoft early next year.

On a more serious note, I hope Van Rossum and Hejlsberg spend lots of quality time together and thoroughly pick each other's brains. This could result in great things in the programming language space.

You don't happen to have anywhere I could read about Stallman potentially joining Microsoft? I find it hard to believe he would.

It was a joke. :-) Hence "sources familiar with their thinking" and "on a more serious note".

Nice, happy to hear that and great for Python. MS has been great for Python.

Nice, big fan.

"Embrace, extend and extinguish" ™

I might need to branch out into other languages.

And the next language to become unusable without a microsoft-controlled ecosystem is...

https://www.youtube.com/watch?v=H27rfr59RiE

What were the previous examples of this?

C#, F#, Typescript, JavaScript via npm

C# and F# are great examples of how MS is willing to allow people out of their ecosystem to use them with all their efforts with .NET Core and having it run places other than Windows.

npm is trivial to replace with yarn and a custom registry.

There were C# and TypeScript outside of MS first?

Embrace, Extend, Extinguish .

Nah that was the old MS, new MS just replaces own with rent.

That is more boring than retirement.

Reply on news.ycombinator.com