Just curious what does a Python team do? It doesn’t make sense to dedicate a team to a programming language.
To take a basic example: Google has a team that works inside LLVM and Clang to improve the code it produces for Google's specific workloads. (Often these help all of LLVM's clients.) But if this team is able to make a search query faster by even 0.01% a year, at Google scale that saves literally millions of hours in compute across Google's fleet. And millions of hours of compute is millions of dollars in power, space, and need to expand.
This is true of all the other language teams at Google too.
The Python team builds a better python by fixing upstream bugs, reducing the memory it consumes and so on. It upgrades the internal installation of Python to the newest versions, ensuring that the upgrade is smooth and doesn' break the using teams.
It contributes proposals for language features to upstream, ensuring that Google's use cases are at least considered.
* maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes
* maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes
* had highly customised versions of tools like pylint and black, targeted to google's style guide and overall codebase
* contributed to pybind11, and maintained tools for c++ integration
* developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine
* developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)
* performed automated refactorings across hundreds of millions of lines of code
and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable.
and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the "marathon, not sprint" approach to work. as i said in another comment, it's the best job i've ever had, and i'll miss it deeply.
we did also have a separate tool that could merge the inferred types back into the source code as annotations, but it never got much uptake.
i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code in.
this was a very real issue for things like the python runtime or widely used code checkers like pytype or pylint, because if e.g. the new version of python, or some improved check in pytype, started raising failures in a code pattern that was technically wrong but which the existing toolchain did not complain about, you could not release the new version until you had fixed all those new breakages.
contrast this with non-monorepo codebases, where the other code would have had to deal with the fact that "oh, our code works under 3.10 but 3.11 broke it, guess we have to pin our own repo to 3.10 until we fix it", but as the python team we could just say "we support 3.11 now, you need to catch up"
Google engineers hate this one weird hack: just send a mail saying "we're deleting/deprecating [everything your product depends on] in [14-180] days, good luck!"
At Microsoft, if there were 12 teams depending on service foo, the team who owned foo had to have a deprecation plan that included sufficient time and support for all 12 teams to get off foo, and directors/VPs would meet and come to terms before the deprecation started.
Google is the opposite. Despite an increasing trend towards top-down direction there's no requirement or support for involving stakeholders. There can be internal products (let's say a data storage solution) and the team who owns it can say "we don't care about this anymore, it's being shut down in 6 months" even if dozens of other teams and millions of end users rely on it, and it's up to those teams to scramble, put other projects on hold, and try to migrate as fast as possible to avoid an outage.
It's frankly demotivating. For the last 18 months, a huge percentage of eng time on my team has gone to either compliance or mandatory migration work i.e. stuff that ABSOLUTE BEST CASE customers don't notice.
> very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)
Just wait a few years. When they come calling for consulting services, charge them triple.
I feel for ya, zem; if you ever turn up at a PyCon in person, lemme buy you a drink.
Several of us were/are/TBD also involved in both long term strategic leadership and maintenance of the open source CPython project itself. That direct feedback line from a major diverse needs user into the project and ecosystem was valuable for the world.
The reason I stayed on this team for 12+ years is as zem said. It was an ideal impactful alignment of people, abilities, priorities, and work life balance. My prior teams at Google... were often not.
For the first half of our Python teams existence, there were only ~5 of us. Many early years were spent paying down internal tech debt accumulated from prior years of neglecting to have a strong Python strategy and letting too many do their own thing. Python was one of the very first languages used widely at Google. It was the last major backend language to get a language team.
Signed, -- the now-ex runtimes TL
I run HR at a small software company reports and my HR person has a lot of friends in HR at FAANGs. For 10+ years we've been talking about hiring and salaries at Google (and others). It's just crazy to me how high salaries have gotten. I know someone that switched from Google to FB when they made $750k at Google and got $1M at FB.
I said all this to say: I've talked to a bunch of people over the last 6 months that know someone laid off and: 1) they had a very high salary, and 2) the group they were in was hiring.
Here is the job description: https://www.getcruise.com/careers/jobs/2831153/
If any of you are interested, feel free to directly reach out to the recruiter Connor Anderson (Connor.Anderson@getcruise.com) or feel free to apply directly.
Hi Zem could you say more about how you maintained those thousands of third party packages? We are facing the same issue and would love to hear your insight!
* copybara [https://github.com/google/copybara] to mirror between github and our internal repo and transform the source tree layout to our internal conventions
* mercurial patch queues [https://hgbook.red-bean.com/read/managing-change-with-mercur...] to apply internal changes
there are internal tools to automate the copybara configuration for python packages specifically, and to map between mercurial and google's version control system, but that's the main idea, and a specific review process for adding new packages.
the process is actually documented here though not all the referenced tools are available (or even applicable) externally: https://opensource.google/documentation/reference/thirdparty...
This is very interesting. Can you share more about this?
one of the current "extremely large python codebase" projects is instagram, and they do have some public repos, notably monkeytype (https://github.com/Instagram/MonkeyType) which youtube did have its own analogue of, and cinder (https://engineering.fb.com/2022/05/02/open-source/cinder-jit...). in general facebook's engineering blog is a great place to read about this sort of thing.