Software Engineering at Google (2020)
abseil.io
abseil.io
A lot of other companies get into trouble trying to cargo-cult what Google does when they are operating in very different environments wherein those practices aren't optimal. E.g. different levels of scale.
Additionally, critics of Google may point out that their engineering culture may not be great on its own terms -- every time Google launches a new feature, people post links to the Google product graveyard.
A lot of the overall goals of cloud are more ambitious than AWS offerings. Reliability is prized more than it is in other areas of Google as well, because customers are so technical and often notice.
Not a place to coast, but I’d say most people do a solid 45 a week for those that want to get good reviews and get a fat bonus.
In what sense?
The problems are a lot more technical though and I don't see a lot of L3s being able to work in the environment as it requires a lot more intuition and experience.
I usually work 45hours a week, but I don't mind it. Plus I'm 100% WFH here cause my management isn't dealing with in office bs.
It is personally scary when they develop new products. What if it is a brilliant idea, one I cannot live without? If Google develops it, then I am looking at this stillborn thing, mewling for life when I know its horrible fate.
The trouble here is that Google employees (and perhaps even its upper management) want to believe that they are a company which is an inventor of things. But they are not this at all. They are an advertisement company. Advertisement companies should not and do not want to invent things... inventions are worse than burdens, inventions are these weird alien objects that appear valuable but are quite expensive and do not help to sell ads at all.
So they hawk the inventions like they were freaks in some carnival sideshow to move traffic past their billboards. Until the traffic dwindles (or until they get tired of it). And then they take it out back behind the woodshed and put an end to it.
The inventions are the small tax they pay to pretend to candidates that they could work on inventions when the vast majority of them will be "allocated" to ads.
Facebook survived because G+ product vision was so out of touch with reality and FB were not part of the anti-compete hiring nonsense so that they managed to poach a lot of good people.
That's fucked-in-the-head just enough that you've made me wonder if it's true.
I'm sure there's some Shiny Invention Corner in the ads business -- let's call it "AI" -- and some of the top people can be motivated to work there.
But isn't the ads business by its very firehose-of-money nature something that will get on fine with that average level of talent that is sufficiently motivated by cash and doesn't need Inventions?
And isn't the top talent able to make the same money doing interesting things elsewhere? (I keep hearing this is happening with AI, but I hear it on Xwitter so who knows.)
Stage 1: "We're not Google, we don't need [[whatever]]";
Stage 2: Foreseeable disaster for which [[whatever]] was intended to address happens;
Stage 3: Giant circus of P0/SEV0 action items while everyone assiduously ignores the [[whatever]];
Stage 4: Quiet accretion, over several years, of the [[whatever]] by people who understand it.
And the [[whatever]] ranges from things that are obviously beneficial like pre-commit code review to other clear winners like multi-tenant machines, unit testing, user data encryption, etc etc. It is an extremely strange industry that fails to study and adopt the ways of their extremely successful competitors.
Now, some of the things you say like unit-testing and user data encryption are ones that I've never seen associated with the "We're not google" mindset, so maybe people have started using that phrase for anythingnow
Pre-commit review is common but not universal in the industry. Some shops practice post-commit review or no review. Some believe review consists entirely of quibbling.
I'm very familiar with CR at other companies, but tbh since most use Git, I wouldn't call that "pre-commit" but "pre-merge", if you will - unless I misunderstood and it really is pre-commit at Google (i.e. the changes are not even _committed_ to the repository - and then I'm confused once again at what exactly that means..)
Edit: as OP mentioned, it does seem to differ in technical sense from traditional CR, in that the changes live only on developer machine, not in source control.
Yes and no. There's a decent whitepaper on Piper and citc you can find by searching for it (or actually I will :P [0]), as far as piper is concerned they aren't checked into source control, but the vast majority of development happens in "citc" workspaces, which aren't source control, but also are source control in a sense that every save is snapshotted and you can do point in time or point in history recovery, and I can open up a readonly view of someone else's workspace if I know the right incantations, and most of the developer tools treat the local citc workspace enough-like a commit that it's transparent ish.
[0]: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
So thats not just a basic feature?
If that sounds somewhat magical, yes, correct.
Which seems worse.
They are actually saved within a special file-system called "citc" (Clients-in-the-cloud). It saves literally every single revision of the file written during development. If you hit save, it is saved in perpetuity, which has saved me a bunch of times. Every single one. No need for any kind of commit or anything else.
Further, these saved revisions are all fully accessible to every engineer within the company, any time they want.
NB: At that company there are also users of git-compatible and hg-compatible tools, but I am discussing the main perforce-derived flow.
It’s quite interesting/mind-bending to think of work-in-progress that’s still somehow synced between peers (in fact this is one of those “missing nice-to-haves” I wish Git had, and can only be approximated with wip branches..)
The SPDK project has an elaborate pre-commit review and test system all in public. See https://spdk.io/development . I wouldn't want to work on a project that doesn't have infrastructure like this.
Even mailing lists with patches are really a pre-commit review system, as are GitHub pull requests. Pre-commit testing seems more elusive though.
Can you name such mid-sized companies with thousands of engineers? If you hit thousands of engineers headcount, you are not a mid-sized anymore.
Theoretically, Google does 100 things right, pays for those 100 things, but also Google has tons of cash, if Google didn't release product in Q1, no worries, they will release in Q3.
Now consider startup with 50 engineers, if you didn't release feature in Q1, you might need to stop the project, because customer with whom you signed the contract just goes away and you will be laying off 5 people
Also sometimes you just need a simple tool to get something done. As engineers we like to build things so sometimes we make it way more complex than it really needs to be. For someone like google that may just be fine to do. For others a minimum viable product may be in order. Do not worry about optimizing for the 3 million user per day case when you have 10 total users a month. Add logging and keep an eye on it. Then worry if you need to scale. As building good scale takes time and thought. Many times you do not need that at all.
As your company/group grows you will take on more and more of the things 'google does' because you will need to, or you will go nuts chasing everything. You could probably even make stages out of the different times to do/evaluate things. To do it early could actually harm what you are doing. You need to evaluate what you are doing and why. Just copying someone else does not always lead to a good outcome and you could be wasting effort when you could be making product.
It's no different to cargo culting. That should not be the reason for not doing something, any more than the opposite should be a reason for doing it. Just see if the practice makes sense in your context and decide that way.
Those kinds of demand happen exactly because they saw them at Google, and an outright refusal is exactly how they should be dealt with. Once the unreasonable person is cut out, you can look at your context and decide what's the best way to solve the problem.
The promo process is not based on value or difficulty, but on the size of the organization that one is running. This is also true for higher level ICs, except they do not manage people, but rather manage/lead projects (which then have a certain amount of people involved).
Here's a rough breakdown:
- L4 -> 1 person
- L5 -> 1-3 people
- L6 -> ~7 people
- L7 -> ~25 people
- L8 -> ~70 people
Misunderstanding this is the source of almost all frustration with the promo process. This process is designed to build and expand the organization, not reward awesomeness. There are of course deviations from the simple schema I listed here, but this is the hard reference point.
But this is not just a Google specific issue, and it is quite widespread in the industry. Google however suffered from this especially due to its obsessive culture of pay-for-perf and by ignoring simple facts:
- inflation means that your salaries should raise regardless of performance. if you only tail the market by adjusting salaries only if the market changes, then you are 1 year late(at least). This isn't a problem in an economy with low inflation, but is a huge problem in one with much higher inflation.
- there is a significant number of people needed to maintain projects that won't show large impact. Those people need to be at the very least recognized and compensated.
- making new products is great, but it requires huge amounts of ressources to do at Google scale from the get-go. A wider strategy is much needed, which Google obviously lacked for almost a decade.
People at L4 might not really like Google's process, but if some Distinguished Engineer shows up at your design review you're pretty much guaranteed to get some sort of valuable feedback. That is not a given in other organizations.
The problem lay with picking the cheapest contractor bids. If anything, the FAANGs should establish consulting arms for this kind of work.
How would I fix it? Not get to the point where the shots are being called by people with 2000 direct reports, for one. Software has distinct diseconomies of scale, where you have a much smaller loop between "Notice a problem. Identify who can solve the problem. Solve the problem" in a small organization than a big one. But that ship has sailed.
Failing that, I think orgs need to adopt quantitative measures of impact (eg. X tickets closed, X customers helped, X new sales generated) along with backpressure mechanisms to ensure that those metrics are legitimate (eg. you can't just create new bugs to solve them; you can't just help customers only for them to need to return tomorrow; you can't generate new sales that are unprofitable).
But this is "some impactful stuff is unrewarded" rather that "hard stuff that isn't impactful is rewarded", which was the complaint in the post above mine.
Good entrepreneurs go into markets where the existing alternatives suck, and make them radically better. Most Google executives and PMs (most executives, period) go into markets that their company and all of their competitors already have pretty decent alternatives for, so the true impact is pretty limited.
As a company grows and matures, their software development processes evolve to meet the business needs.
1. Money. Google essentially has a giant, gargantuan, enormous, bottomless pit of money to build a lot of this tooling (and also to take the risk if something ends up not working out). I think you might be able to say that other companies are just being short sighted if they don't implement some of these things up front, and that may be true, but (a) that's pretty much human nature, and (b) given that very few other companies have a bottomless pit of money like Google, that may just end up being the right decision (i.e. survive now and deal with the pain later).
2. Talent. This is closely related to #1, but few other companies have the engineering talent that Google does. If there is one thing I've seen with my experience with ex-Googlers is that most of them are fast coders. So when you go to your boss and say "I'd like to implement engineering/tech-debt improvement XYZ", at other companies it's a harder decision if (on average) it would take 9 months to implement vs. 2 or 3.
3. Related to both of the above, but your 4th bullet point, "Quiet accretion, over several years, of the [[whatever]] by people who understand it.", is actually other companies just waiting for more evidence to see what "shakes out" as the industry-standard, optimal way to do things.
4. Finally, your stage 1, "We're not Google, we don't need [[whatever]]" is actually true in tons of cases. Many of Google's processes are there to handle enormous scale, both in terms of their application/data capacity, as well as the sheer number of engineers they need to coordinate. Very, very, very few companies will ever hit Google's scale.
E.g. things like "do feature work on a branch and then code review/run PR checks before merge", "have unit test coverage (being a hard balance to judge what is sufficient coverage)", "have useful comments" - absolutely none of these things I associate with "Google engineering practices", and many of them definitely predate things that were specifically done at Google.
Things I think of when I think about Google practices are things like ensuring data is infinitely horizontally scalable, monorepos, etc. Those things are all scale-specific.
I usually see the pressure to split into microservices appear around 20 engineers, just as your single repo is starting to get unwieldy. Knowing that the big companies use a monorepo is pretty important information here, because it may prompt you to invest in tooling to make that one repo less unwieldy rather than splitting into many small repos that will be very difficult if not impossible to merge back together again.
Google doesn't actually plan for infinite horizontal scalability in data. The framing I've found most useful is [Jeff] Dean's Law: "Plan for a 10x increase in scale, but accept that sometime before you reach 100x, you will have to rewrite the system with a different architecture." The reason for this is shifting bottlenecks: as the system gets larger, different aspects become the bottleneck to future scalability, and each time the bottleneck changes you usually need a different architecture. But by planning for an order of magnitude growth, you ensure that you're not artificially introducing bottlenecks, and that you have enough headroom to actually discover the new bottleneck.
A monorepo really doesn't have anything to do with how code components are deployed - your comment seemed to be contrasting a monolith architecture with a microservices one.
It's impractical (particularly when the project is young) for the same reason having separate binaries is impractical: it makes it very difficult to do refactorings that cut across repos while still keeping atomic checkouts and rollbacks.
1. When you're small enough, and only have a few teams and/or deployed binaries, you keep everything in a single repo.
2. As you grow with more teams and more products/binaries, often companies will split into having separate repos per team/product, and then use some sort of dependency management tooling (either something like the git submodules you discuss, or a private package repository, e.g GitHub's Package Registry, Nexus, or Artifactory). I totally agree with you that a lot of companies do this prematurely.
3. What distinguishes "monorepos", in my opinion, is that this was an innovation at either Google or Facebook I think (not sure who was first), where they realized exactly what you point out - it makes it really hard to do refactorings across lots of dependent projects in different repos. So they decided to keep everything in a single source control repository, with a single commit history. But in order to do that and have things be sane with lots of teams and thousands of developers, they needed to invest a ton in custom tooling to make these giant monorepos usable by individual developers and teams. E.g Facebook has Sapling, Google has Piper, and there are open source tools like Lerna for JS monorepos.
So, in my experience, just having everything in a single repository but without any special tooling (because you're small enough to not need it yet) is just a repo. Monorepo IMO implies that you've grown to the point where it's difficult to keep everything in a single repo without special developer workflow support tools.
All that said, I definitely agree with your main point - a lot of companies can just keep everything in a single repository a lot longer than they think they can, even without special tooling beyond some separate build targets.
1: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
With these companies I think it's just best to walk the other way. Some of them may state that "yeah, we know we need to improve our engineering practices, so we're willing to learn" but they usually just have so much of a different mindset of what it takes to actually run a software company that it's just a waste of time. There are many more companies that have a modicum of understanding about what they're doing.
But there's also the bad: Software isn't the company's focus, so you aren't the company's focus. That means no "Senior FAANG" salaries, no best practices to keep things sane, and often you find yourself working on a codebase that was originally hacked together in a week by a chemist who may or may not have been deliberating huffing reagent fumes.
Being Google means having a team which writes source control management system for you.
More and more companies will be like this to cut costs
What ex-googlers often fail to grapple with is the product lifecycle (how short it may be) and the value of having diversity in the loop of product testing. Google is designed to be a safe place to focus, and that’s not what the real world is like outside the plex.
The fact that Plus and Glass got canceled and Wallet has been canceled sixteen different times is merely a consequence of the fact that leadership and product is often led by imbeciles. That's an organizational problem and I hope nobody is out there cargo-culting Google's org (even though I know they are, with OKRs and Perf being widely copied).
Everything else isn't. Unit tests aren't a luxury that Google's infinite riches allow it to have - they pay dividends whenever code exists for more than a few weeks.
You can bet your ass Google engineers don't write unit tests for throwaway code.
CI saves time, and while Google can maintain a team you can afford to pay for Jenkins or GitHub Actions, because not paying for them is more expensive - if your company is to survive for more than 3 months.
A CI bot which waits 24 hours then says "no", with a text file that crashes your browser and ultimately only contains the information 'exit code nonzero', which fails for reasons totally unrelated to your code change is dubious as a value add system.
If that bot is also a non-negotiable gate on shipping things you get a bunch of other antipatterns, like massive code patches to decrease how often you have to roll the die and a tendency to hit retry every day or so until the probability that it's actually your patch that's broken gets high enough that you try to debug it locally, at which point you may be unable to reproduce the blocking error anyway.
The real question is whether that pathologically rubbish implementation is still better than shipping without CI, which rather depends on whether your engineers ship code that works without the guide rails, which to a fair approximation they do not.
Thus it might still be a net win for product quality but saving time is harder to see.
Of course, the difficult part about managing a CI pipeline is writing quality tests, ensuring your tests don't take forever, deciding the right balance between mocking out 3rd party runtime service dependencies vs. calling their dev versions in your tests, etc.
But this is why I argue that the bare minimum should just be to have the CI pipeline created. If you don't have that, you are definitely going to screw over future you. Once that's there you can balance the cost/benefits of how much to invest in your test suite and test coverage.
Eg - most telco companies in the US run at a scale similar to google. They need most of the software engineering best practices, internal tools teams, etc. They used to have it during Ma Bell times when they had a cash cow. That doesn’t exist and they’re left with scale and the points 1-4 described above.
This in general leads them to outsource to lowest bidder contracting firms that compound the shitty software problem. In the end it’s a miracle that all of it works together :)
Unless you're referring to automated precommit hooks, this sounds baffling. What's wrong with reviewing pull requests? What if I want to push a WIP while I switch to another branch, I still need a review? Is the final PR reviewed again at the end?
Anyway, TL;DR, the problems you suggest are git-specific and one solution to them is not using git.
I'll give you "pushed the WEB industry to have transport-layer encryption for the entire industry by default".
I'll even give you "code reviews".
But not the first 3.
Google has (or had ~10 years ago), a thing called "readability" for each language, where in order to be allowed to commit code to the central "google3" repo, you needed to have written some large amount of code in that language, and needed to have a readability reviewer sign off on your code. The process is designed for slowly on-boarding junior people into a team, and introducing them to google coding style and practices. Eg, the senior, mentoring folks on the team do the reviews and bring the new person up to speed. I imagine it must work well in that context.
However, this breaks down when your entire team is new. How do you find somebody to review the code? All several million lines of the product that was acquired? Especially when it is written in multiple languages.
So we were basically locked out of the main corporate repo, unable to do anything productive. We finally figured out that there was a paved path with a git repo used by the kernel team (and android?) that had none of these hurdles, where we could put our code and get productive immediately.
(Left Google a year ago)
There's a form to get your corner of the codebase exempted from readability temporarily. This gives your team a quarter or two to build up readability.
Old code can usually be left as-is unless there some particularly egregious security hole or the like.
My previous team - pretty much any python submission was hitting me with a python "readability" requirement, and it was a bit painful, because only a single person in my entire group of teams (roughly 15 people total) had the "python readability expert" status. My current team - already submitted quite a few significant C++/TS/Java pieces of code to G3, and not a single "readability" requirement triggered.
And there is more to style than just linting, IMO. For example in C++ there are some complex macro-based test predicates that are hard to learn and use but which greatly simplify/improve on naive testing. Part of the point of C++ readability is that people who understand this stuff teach new people how to use them, or at least introduce them to concept, during code review
Exactly. It is very likely for a lot of junior engineers will be working with other junior engineers, and they will in fact have the most specific knowledge of the part of the project that they are implementing. And human nature makes it so people are afraid of being judged by their "superiors". Readability makes so that it breaks that barrier, guarantees that a more senior engineer will be involved, and will teach the ropes into writing readable, mantainable code to nooglers.
I was a readability reviewer and most of the readability CLs were the first project a person worked on at Google, often rather unnecessary but redone strictly to meet readability requirements (largely new code, more than X lines, etc.). I would go back and forth for quite a while to turn 1000 careless lines of throwaway code into 50 lines that were actually good, but I basically had to grant readability after that one interaction, and it never felt great to me.
The most hated readability process at Google was Go's process (at least in the early days; k8s is obviously not using it), but I think it was actually one of the best. It took me a long time to get Go readability, but after going through the process I feel like I'd write the same Go code as anyone on the Go team. When I look at people's open source projects I think to myself "don't they know that that Simply Isn't Done?" But of course they don't; Go readability can only be experienced, not explained. People didn't like that process, and I am sure I said nasty things about it at the time, but in retrospect I really like it.
After taking a walk and reflecting, though, I'm remembering something that my manager said to me when I gave notice. Google is not for everyone, for a lot of reasons, and especially a lot of people who came up in startups really have problems (which is ironic since so many startups come out of people leaving Google). How you feel about readability may actually be a pretty good test of whether you will fit in at Google in the current era: it's not a small, scrappy company anymore that gets shit done quickly using whatever tool is most efficient RIGHT NOW and ships it as fast as possible to see if it gets product/market fit. It's a behemoth that runs one of the most prolific money-printing machines that has ever been built, and fucking that up would be a DISASTER. It'd be better to have half the engineers at the company do literally nothing for 10 out of 12 months in the year than to let someone accidentally break the money-printing machine for a day while they figure out how to fix it.
And obviously, it's better if everyone is productive even as they're shuffled around from project to project (which they will be, a lot), which means that you want as little "voice" as possible in their code. At a lot of companies you can tell exactly who wrote a line of code just by the style (naming, patterns, etc.), without even checking git blame, but at a place like Google individual styles cause problems. So the goal is to erase as much individual voice/style/preference as possible, and make sure that anyone can slot in and take over at any point, without having to bother the person that originally wrote the code to explain it (they might be at another project, another division, another company, and even if they're still at Google there is a very strong sense that once a handoff is complete, you should not be bugging people to provide support for stuff they've moved on from).
In that sense coding at Google is a lot closer to singing in a choir than being the frontman in a band: you need to commit to and be happy with minimizing what makes you unique or quirky, rather than trying to accentuate it and stand out. Some top-tier singers just can't force their vibratos down, or hide their distinctive timbres, or learn to blend with a group, and are absolute trash in a choir; it's not their fault or some ego failure, it's just that there are some voice types that don't work in groups, and that's fine, you just don't add those people to a choir.
At least below director-level (or maybe L7 equivalent on an IC track), Google doesn't need individuals to come in and shake things up, bust apart processes and "10x" a codebase. That's startup shit, and even if it might sometimes be worth some risk for the high payoff, it's too dangerous for them to allow for the thousands upon thousands of (still quite senior, sometimes 15+ years of experience) L4 or L5s at the company. The same process that prevents that from happening also makes sure that the entire machine keeps humming along smoothly. If being a part of that smoothly functioning machine while painting within the lines is exciting, then Google can be one of the best places on the planet to work; if you would be driven crazy because you can't flex and YOLO a prototype out to users in a couple days, then it's really not going to be for you.
I'm in the latter camp, I couldn't handle almost anything about the process and was so desperate to move quickly that I started talking to investors to line up my own funding a few months after I joined, but even as a quick-quit (<1 year), I have the utmost respect for the company and the people, and highly recommend it to almost everyone who applies there (the exception being people like me who TBH should just be doing their own startups). Everything they do has a pretty well thought out reason, even if I don't like following those rules myself.
DISCLAIMER 2: I think from a hands-on-keyboard SWE there is a lot of useful stuff. What you mentioned about Google culture of killing products and such I am not gonna talk about.
I recommend chapters about testing first and foremost. Among all the codebases I saw (both OS and proprietary) Google tests are the most comprehensive and reliable. However, If you are in a startup-like environment you should pick and choose and not try to follow every single principle listed as they could sink your velocity drastically in the short run.
Other interesting points (IMHO) are Monorepo, Build System, and Code Reviews.
For the Monorepo I discover being a huge lover although I was skeptical. The sad thing is that it's a rather niche practice and tools like Git don't play ball very well (i.e. each time you pull you have to retrieve changes for all the codebase, even files you never saw/heard of managed by another team). I think there's no nice off-the-shelf offering for running monorepos out there. However, not having to fight with git submodules, library versions, ... is great. If the change I am submitting breaks something else in the company you are immediately aware and so can act accordingly (e.g. keep the old implementation alongside the new one and mark it as deprecated so the other team will get a warning next time they do anything).
The build system is a bit more controversial. I learned to love blaze/bazel, but admittedly, the OS version is a bit messy to set up. Additionally, being so rigorous about the build rules felt like a massive chore at the beginning, but now I appreciate it a lot. I can instantly know the contacts of all the teams that use a build rule I declared and hence can be contacted to warn them about bugs, ... . I can create something experimental and have private visibility so only my team can use it and only later expose it to the wider world with just a one liner.
Finally, the code review AKA Critique. Google has the best review tool I had the joy to use hands down. It's clear about what happens, at which stage is the review of a particular section/file and is focused on discussion. The evolution of each change is easy to follow along. These are things I really miss when using GitHub/GitLab PR view. The tooling is incredibly confusing to me. Luckily (I am not affiliated in any way) an ex-Googler (I believe) is working on an alternative that works with GitHub (https://codeapprove.com/).
I think Git works perfectly well for 99% of monorepos though. It just doesn’t work for the massive ones. I think its a perfect example of something most codebases shouldn’t follow google on.
Again, this is really only a problem at larger scale. The segmented number of people that pull this large blob at Google may be higher than the entire engineering team at other companies.
Thing is, pkg-config would've picked up the dependencies just fine - they were literally all there. I even built Rust from source on my weird machine with musl variant, before realising musl has some issues on my architecture.
I suspect Bazel may work well inside of Google for infra/server side stuff (never worked there). I'm a lot more skeptical of more complex builds, like desktop applications across various platforms. Chrome still uses "gn" to generate ninja files, and then ninja to build. For my own stuff, I won't touch it.
I probably wouldn't have commented, except that, to my surprise it seems the Anki developers have also decided life is too short https://github.com/ankitects/anki/commit/5e0a761b875fff4c9e4...
In the open source world I see a fair amount of "tests are always red, don't worry" and "we can never edit this interface because who knows who it breaks." These problems aren't intractable at Google.
This approach does have its own set of challenges and I do suspect that the monorepo has contributed in some ways to Google's inability or refusal to maintain some older products. But holy cow the ability to do something like move everybody in the company to different vocabulary types is powerful.
Google's emphasis has always been to make things easy for library developers, at the expense of library clients. For people who value backwards compatibility over long timespans, Google's practices could be better.
And of course volunteers working on open source projects have lower standards. Lets instead compare Google to companies which say “we arent google.”
TBH most of this stuff is transferrable and even "common sense" in most of the companies you've worked for. Similarly how Google's SRE book is actually a very good collection of battle won experience on how ops can keep systems more reliable and running.
The book is written in a way that you can easily throw away advice that you don't think useful.
There are plenty of policies floating around that don’t scale well, and plenty of migrations that are still forced on internal users rather than handled magically by the migrating team. The reality is that Google is such a big company most of these fly under the radar of whichever person actually enforces these policies, and it becomes a whole thing to escalate up to whoever enforced them, and then there’s potentially a political battle between whatever director or VP is in charge of the bad actors and the enforcer (ideally they get away with not allocating HC to the internal migration and amortize it across all their users, so that HC can work on flashier stuff).
I think one reason Google has a proliferation of bureaucracy and red tape is that they do not “review” postmortem action items very formally. They are only reviewed as part of the larger incident postmortem review process and the tooling is way overengineered such that performing that review beyond a perfunctory once over isn’t easy to do. So you end up in a situation where “we need to do something” and whichever person handled the incident has to suggest a way to make sure it doesn’t happen again - the easiest of which is to introduce some CYA process. The other reason is that non-coding EMs introduce processes to show some kind of impact on their team.
Also, the existence of the monorepo, global test runs, forced migrations, etc makes it so maintaining a mothballed project incurs some inherent engineering costs - IMO it’s a non-negligible reason Google kills products that could instead simply exist without changes. It also makes it so Google doesn’t really “version” software generally speaking.
I don't think there is a visible distinction between those parts that work/doesn't work. In fact, most of the cases each practice has pretty strong rationales. The problem is, when you take everything as whole, its cumulative complexity and cognitive overhead tends to go wild and almost no one can understand the whole stack when its original writers/maintainers leave the team.
In fact, this might play a certain role of the Google graveyard narrative; it's not because its engineering culture is bad, but sometime its standard is too high for many cases so it's nearly impossible to keep it up for newcomers, especially when you have external pressures that you cannot ignore. Even if you make an eng team of 3~4 people for a small product, they'll likely suffer from tens of migration/deprecation/mandates over years.
I would say the vast majority of it works well, some you just don't need until you hit scale (here, scale in the number of developers).
For example, policies work if you have <20 engineers, probably don't really work otherwise.
Blaze/Bazel I miss a lot. Just wrangling the dependencies between shared packages is a mess (though we might just suck at configuring Poetry - at any rate it's not intuitive). Building and deploying is much more involved.
Another thing I miss is code review the Google way. Google asks that you review within 24 hours, reviews by (the equivalent of a) commit and not by PR, and strongly advises to keep commits small; The GitHub PR workflow is terrible in comparison:
1) it nudges you into batching commits into large PRs
2) Is the PR message informative? Is each commit's? What about squash and merge - how many people edit that message? At Google part of the code review is reviewing the commit message. When you squash and merge, that's post approval, so you can't even do that.
3) Hidden conversations? What the actual fudge
4) How many comments have I not addressed yet? For that matter, how many PRs are waiting for my attention and when were they sent?
I think coupling commits with review progress was a mistake.
However I think the biggest issue with the landscape for code review tools is that 99% of developers use the default system that ships with their VCS. So on most teams, that's GitHub. People should be actively choosing their code review tools just like they choose their VCS, IDE, CI/CD platform, Issue Trackers, etc. It's one of many tools that makes up your SDLC "Stack".
As long as CodeApprove and Reviewable remain niche they've got nothing to worry about.
It supports some cool things Critique doesn't/didn't, such as reviewing multi-commit branches (also across history-rewriting force-push cleanups), and indicating exactly the nature of your comment (just FYI, or you want this to be changed before you'll give your approval).
(I was an intern in the initial making of Critique, and subsequently got interested in finding an out-of-Google alternative. I contributed a bit to other review tools such as ReviewBoard, and actively used Gerrit, Crucible, Phabricator, and GitLab reviews.)
When looking for a Critique alternative for my startup, reviewable.io had just appeared and ticked all the boxes, and we use it successfully for many years. The drawbacks are that it's GitHub-only and isn't free software.
Re not that much to review, it's the exact opposite. When people get 400 line code reviews they tend to nitpick on style. When they get a 100 line CR, they critique on naming, organization, consistency.
Not "worst" since that would be too hurtful. The hope was to recognize excellence, motivate people to be better, and maybe shame the people whose tools received no votes. This suggestion was summarily dismissed.
There were, indeed, some truly excellent tools: Dremel comes to mind. And lots of tools that were nearly unusable.
Any prominent examples?
Yes, if you have a mountain of money and a horde of underutilized employees, it’s easy to gold plate your engineering and navel-gaze at your biases.
I'm also reminded of something written by Scott Adams (decades ago, when his reputation was different.)
> Most people won't admit how they got their current jobs unless you push them up against a built-in wall unit and punch them in the stomach until they spill their drink and start yelling, "I'LL NEVER INVITE YOU TO ONE OF MY PARTIES AGAIN, YOU DRUNKEN FOOL!"
> I think the reason these annoying people won't tell me how they got their jobs is because they are embarrassed to admit luck was involved. I can't blame them. Typically, the pre-luck part of their careers involved doing something enormously pathetic.
-- "The Dilbert Future" (1997), by Scott Adams
* reproducible fast dev environments - anyone should be able to build anything pretty easily fairly quickly
* culture of design reviews, testing, and code reviews.
* CI/CD, static analysis, PaaS, dependency management.
You can do all of these without Google-level or really any bespoke tooling. A lot of what google builds is for operating at Google scale - distributed builds and running of huge applications - and even then this tooling this simply enables working at this scale, it comes with a lot of cost(speed/complexity) and jank you don't need at smaller software shops.
https://news.ycombinator.com/item?id=31224545 (304 points, 179 comments)
https://news.ycombinator.com/item?id=22609807 (222 points, 70 comments)
Software Engineering at Google (2020) [pdf] - https://news.ycombinator.com/item?id=31224545 - May 2022 (178 comments)
Software Engineering at Google - https://news.ycombinator.com/item?id=22609807 - March 2020 (69 comments)
Similar sounding but different:
Software Engineering at Google (2017) - https://news.ycombinator.com/item?id=18818412 - Jan 2019 (309 comments)
Software Engineering at Google - https://news.ycombinator.com/item?id=13619378 - Feb 2017 (156 comments)
A lot of Google’s practices really are good software engineering practices - provided you have the money to invest in replicating it to a high degree of quality, which could be substantial and better spent elsewhere. When you have one of the most lucrative business models of all time you definitely have the money to invest in trying to make it as stable to maintain and easy to add value onto as possible, so it was definitely worth it for Google in many cases, but each other company will have to determine the costs vs benefits themselves.
Replicating Blaze and Forge seems really expensive and hard to get right (though it can be tremendously valuable for development on a large codebase). Postmortems, containerization, servers-as-cattle, gradual non-global releases… those aren’t as expensive to set up and have great cost/benefit ratios. It’d be stupid to not do these just because Google does them (and in some cases invented or popularized the practice).
This is the guy who built the hotspot JIT that added decades to the life of Java, and who engineered Google's data centers and GCP. He obviously doesn't need more money or glory or experience, so ... why?
There are a million examples of things gone wrong, but it may be worth studying one example of how someone could have such an impact and still just love what he's doing.
He’s not an author and not in the acknowledgments: https://abseil.io/resources/swe-book/html/pr01.html#_acknowl...
I wish Google would relax their 'guidelines' when it comes to software that's also published outside of Google. Case in point: the Dawn C++ library (Google's native WebGPU implementation) has a dependency on abseil, and from what I've seen when glancing over the code, the only reason seems to be some minor string-related stuff.
I can only assume that there must be some interal NIH rule inside Google to use abseil in place of the C++ stdlib (of course I would prefer if Dawn would use neither abseil nor the stdlib, especially since it looks like the only component that's used are related to strings, which definitely isn't the focus of a 3D API).
...and then there's of course the use of 'Google Depot Tools' and of course they are using their own build system [1] (however at least there the Dawn team rebelled and also provides cmake build files).
All those Google specifics make it increadibly difficult to integrate Google C++ projects into any non-Google project, and because of this "Google C++ bubble" I would seriously hesitate taking any advice from them about software engineering as gospel, at least when it comes to C++.
[1] https://chromium.googlesource.com/chromium/src/tools/gn/+/48...
E.g. absl::StrFormat — no equivalent until std::format was standardized in C++20.
absl::StrCat: You could use streams, but, ew. Also, StrCat is optimized to reserve sufficient size in advance, so it is more efficient than appending to a string or using a std::stringstream.
absl::Cord. No equivalent in the stdlib.
It would be tremendously beneficial if their software dropped the abseil dependency, especially where it is almost entirely unused. Hell it'd be better if they simply vendored the bits they need.
Having to use Bazel, and having to manage an additional dependency like abseil can be hellish for small projects with uncomplicated build systems.
The worst part is that abseil leaks through interfaces and you end up being coupled to it as a consumer of a library. It's bananas. I don't need yet another stdlib.
FWIW my understanding is that this is exactly backwards: Abseil exists because the internal code and toolchain evolved to use features that hadn't landed in the standard yet, and releasing the support this way allows that dependent code to be used in open source releases. It's not about features Googler's "can't" use, it's that they[1] could always use better stuff, and this is a way to get the better stuff released so non-Googlers could use it at all (and then, apparently, complain about it).
Obviously looking at this in hindsight from the outside of a project using gcc13/clang15, it seems like it's needlessly different. But when written it was forward-looking.
[1] "We", I guess, though I work in ChromeOS and not in this world.
The dependency here almost certainly predates C++ adopting the features it has had for a decade.
In contrast Facebook Folly has much less overall clout because engineers there have more incentive to build-from-scratch, which can include simply not using C++.
I remember Sanjay Ghemawat or Jeff Dean mentioning that one of their big "optimizations" was to inline short strings into the string object- instead of a string that was "size_t len, size_t capacity, char *data", anything less than 24 bytes was just stored directly instead of with a pointer. when you're running mapreduces with trillions of small keys, this makes a big difference!
All the adages listed here make perfect sense, but they don't succeed in a vacuum. Servant leadership is great, until individuals disagree with product strategy and priorities. Addressing low performers is great, until a company's HR policies make doing so onerous.
I could go on, but there are often many things outside of your control as a leader that directly affect your capability to manage. It's all part of the job and something to account for, but set your expectations accordingly.
I'm of the opinion that both are needed, but don't put all your eggs in unit tests (they don't need to be perfect). That extra time being spent on integration tests tend to be better for maintaining system health.
Also, it's free. [0]
[0] https://www.audible.com/pd/Software-Engineering-at-Google-Au...
I'm not really sure what to make out of these discussions about how X or Y Google engineering is, while the production service is broken for an end user like me.
My guess is that between SEO companies and Google just trying to maximize ad profits the product is in terminal decline
There's no news of a widespread Google failure. Maybe you have a browser extension interfering? Or there's some kind of very localized hiccup.
In any case, your experience right now isn't even close to representative. For its scale and complexity, Google search is probably one of the most reliable services ever built.
As for why this happens, I have no idea. I've had Google Maps completely black out on me and then eventually magically fix itself many months later.
As for reliability, I would probably have agreed if it was a "simple" system (which the original Google was). Today, I'm not so sure. I at least understand that Google today is made up of a large number of subsystems, and subsystem failures like the ones I'm experiencing (and bad search results as others have also reported) do in fact erode my trust in the product. "Your 99.9% is not my 99.9%" feels like an apt quote here.
currently logged on a google account, indeed the "google stock" search shows "Your search - google stock - did not match any documents"
it happens with other searches, too; not all of them, but some.
no solution found at the moment except logging on another account / not using an account. no extensions installed either.
I've had multiple occasions where Google reproducibly fails to find exact matches in the page title (no problem for Bing). This cannot be explained by mysterious AI ranking or Unicode issues since Google gave me zero results, the website is non-political, and the title is just plain ASCII.
This never happened ten years ago. Whatever they are doing now, they are seriously screwing things up.
The query [ google stock ] would never return zero results.
If this is really happening to you, please post an actual screenshot demonstrating that a standard browser in incognito (not logged in) mode on a standard OS returns no results for [ google stock ].
I'm not saying Google's core product hasn't slipped but that's one query I run every day.
I can't tell, right offhand, but is this an official Google publication?
I've seen people rapidly shift to career protection, wagon-circling, and empire building. I've seen more competition for leading junior people since there are fewer new hires coming in. But mostly I've seen the company become more cynical towards executives.
Depending on your feelings on AI, you might see new excitement and opportunity opening up or further meddling and messy product management in the future. I'm not sure where I land here yet.
I think Google is still a very good place to work. Pay is high. WLB is great (at least where I sit). Tooling is very good. I haven't had any asshole managers or directors. But I definitely find it hard to get people excited about taking a big risk or maintain a project that is important but somehow misaligned with what your VP cares about.
> It introduced a perverse incentive: become an important project and your software engineers won’t need to write documents. Discouraging engineers from writing documents turns out to be the opposite of what you want to do. Because they are a limited resource, technical writers should generally focus on tasks that software engineers don’t need to do as part of their normal duties. Usually, this involves writing documents that cross API boundaries. Project Foo might clearly know what documentation Project Foo needs, but it probably has a less clear idea what Project Bar needs. A technical writer is better able to stand in as a person unfamiliar with the domain. In fact, it’s one of their critical roles: to challenge the assumptions your team makes about the utility of your project. It’s one of the reasons why many, if not most, software engineering technical writers tend to focus on this specific type of API documentation.
PMs are a scarce resource. A lot of eng teams within Google don't have a dedicated PM and need to carve out product-market fit for themselves. When you have a dedicated PM it's easier to offload all PM ideas and responsibilities to that person.
(I'm a technical writer at Google. The boilerplate "all opinions my own" is important in this convo because I think a lot of TWs and PMs will strongly disagree with these ideas.)
Maybe it’s more of a PM culture thing.
But as a dev (not at Google) I’m used to stepping into self-driven mode when PMs are slacking, and it’s a shame for Google that Googlers don’t exhibit this behavior.
Google is so much not like any other company, especially 20 employee startups. Trying to apply their recipes to 99% of companies is solving problems that don't exist and most likely creating new ones you don't need to deal with.
Microsoft is a much better example for business software as they are (were?) paranoid about backward compatibility.
For example, engineering usually tells product, if you change feature A, then it will also affect feature B, C and Z. Otherwise you may end up with contract breaches and SLA violations.
Product lifetime and providing incremental features is a big reason why SAP and Oracle have been successful in the enterprise space and people still pay a lot of money to buy them.
Yes, but are the well-engineered software?
Testing, reproducibility, etc. These have always been common sense. It's just that people only adopt them when the big guys do it because we implicitly give authority to those with money and power.
It's not different than when you see folks making videos of "How a millionaire structures his time." Like, wtf. Those ideas existed before the millionaire. Many people already practice those. And those alone are not what made the millionaire.
If you have a decade in control of the greatest internet business ever, go ahead and do exactly as Google has done. For anyone else, look inward for inspiration!
I havent really see Google doing any innovation in about a decade or more? What have they done significant since maps and android over a decade ago?
Apple stomping them in hardware, openai stomping them in AI, AWS stomping them in cloud, Nvidia stomping them on game streaming.
Google has a monopoly on a big ad network at its core and thats not high performing or innovative.
This is an html copy of a physical book. In book, that's nothing unusual.
The formatting of the table of contents does not convey it well.
...and then it's blank.
Am I missing something?
(I bought it, intending it as interview prep for an experienced engineer interview, but seems they are doing student-startup-style Leetcode interviews.)
Is this really the case? In high school I sure thought Google was a magical software heaven us mortals could but dream of working for, but now (and increasingly as of late) I'm strongly under the impression that 20 years ago they discovered unobtanium in the form of a very under-served search and ads market, they plopped down camp on top of it and turned it into a giant money firehose, and now to this day are simply operating that ridiculously high-margin business while trying desperately to find another vein of unobtanium because when you already have one, that's the only thing that even registers on your radar.
"Revenue from Google Network ads hit $7.5 billion (10.7%), and YouTube ad revenues registered at $6.7 billion (9.6%). In other words, at $54.5 billion, the total ad revenue from Google constituted 78.2% of the company's total revenue in the quarter." https://www.oberlo.com/statistics/how-does-google-make-money
78% between google and youtube. And given youtube isn't nearly as high-margin as the search product (can you even imagine the cost related to essentially storing the video history of the world?), I'm hesitant to say youtube is their 2nd vein.
So yeah. Turns out when you have essentially a monopoly in a high-margin business, you can do basically anything and build an entire private software ecosystem without going bankrupt.
I interpret "how google does it" to simply mean "look at all this cool stuff you can do when your product is this high-margin".
N.b., I’m not saying nosefrog is this person, or even everyone at google or ex-google is like this. This was just my experience working with senior principal or other very senior engineering staff.
In 2010 when I interned there, one of my mentors said something to me that stuck with me: "Google found a hose that money pours out of, and it's name is 'online advertising'. All we do is optimize that hose, and search for another one."
Sonos: https://www.theverge.com/2023/5/26/23739273/google-sonos-sma...
A MIT student: https://news.ycombinator.com/item?id=18566929
I met them (they were horribly unprofessional) in 2013 around time Sonos did to discuss my audio syncing tech. Best to steal tech then bother investing into R&D. Quickest way to make more money .. steal .. spend less and make more.
I haven't mentioned my experience in years cause it's old news but Sonos court victory is new relevant news to my experience meeting Google R&D that same year.
Though I can say for sure and claim is that they were extremely and laughably unprofessional to me and I met with other tech companies then who were very professional and polite.
I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product.
When I was younger, I wanted so badly, to work at Apple.
A few years ago, they actually approached me, and I found out that the culture seems to have drastically changed.
They ended up not wanting me anyway, so maybe it's just sour grapes, on my part.
Can you explain in detail?
But I don't think that someone like me would be very welcome there.
That is both good and bad. I am certainly not God's Gift to Programming, so they may well be better off without folks like me. They certainly seem to be making a lot of green.
Wouldn't it be helpful to other people who are interested in working there?
This is a professional venue, and I don’t feel that slagging companies is a particularly professional thing to do. I like Apple, and wish them the best. I have no interest in throwing shade on them.
When I do say less-than-positive things, I find it best to stay vague, so people can easily write off what I say. I think that folks here, are perfectly capable of making their own decisions, and that I’m best off, letting them do so, unencumbered by my opinions.
But the software side? There seems to be a massive variance in both ability, and giving-a-shit across the software orgs, and no leadership filter on the output of these disparate groups.
Perhaps they have the opposite of the resting-on-laurels problem: a smattering of very senior people who are talented and care about the product/users, and a wide base of careerists who aren't very good and just want the money and the CV, users be damned.
I was told about e.g. multiple internal CI/build systems because one org didn't want to rely on the other, so they both built their own little software stacks.
That's a bit different than what I've heard from meta/google/amazon who seem to be more open to running centralized internal services
https://gist.github.com/chitchcock/1281611
This was rather a long time ago now so I wouldn't be surprised if things have changed since then, though having never worked at Amazon I wouldn't know one way or the other.
The obvious thing to do then is to not change too many things around it. Or at least document the changes.
So then you get a bunch of processes that are designed to not rock the boat too much, in case the magic dies.
I don't know if the process is more important, but at a certain size codified process becomes a requirement.
> I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product.
Why change it if it's working well? Keep doing what you are good at, keep being the best in the world/market at this, and the customers will reward you.
I seriously don't understand the common theme here in these threads saying that Google is basically stagnating. Why do they need to churn out a major new product every other year to stay relevant? They continuously re-vamp their entire stack, their data centers, their networks. Nothing ever stays as it is. As a user I actually find that things change too much. (But I'm not what counts, it's ad revenue that counts..)
Generally programmers at Google are better than programmers I’ve worked with outside of the company. But not by leaps and bounds. And in my opinion it doesn’t matter as your pace of development is so much slower it matters little how good of a programmer you are.
Both can be true at the same time, and my feeling is that they are, having worked there. The business is not very diversified, and the tools are great.
That wasn't the argument. The argument was simply that having a profitable monopoly doesn't imply the presence of quality.
Google's massive internal processes and systems are the result of having already been huge for many years. Google needed the time and resources to build them up. They're not the cause of Google's becoming huge in the first place.
Of course the quality of Google's external search results helped Google achieve a monopoly. Ironically, many people say that Google's search results have been getting worse, and I'd have to agree with that sentiment.
See also the list of mergers and acquisitions by Alphabet: https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitio...
I've give one example I observed commonly early in my career. The unexplained memory leak. Some process is running and the memory usage continues to grow. Eventually it will use the entire memory of its machine and die. You have a few options: 1) debug the issue and address the root cause 2) debug the issue and workaround it in some way 3) Give up and rewrite the code using some other kind of tooling 4) wake up people when the process dies and have them restart it 5) write a cron-job that restarts the process periodically.
What is the right answer? The best from a QA perspective is probably to identify the root cause and fix the underlying issue. Tools like valgrind have made this much easier in recent years, but it still can be a challenge. Pragmatically, my own answer (speaking generally, there are more different cases for different contexts) would be to time-box and investigation and fix, and if that wasn't achievable in reasonable time, just write the cron-job and work on the next problem. You can imagine very successful operations filled with kludges like that. Is that low quality?
But I can't think of anything like that in the last 15 years.
I had a front-row seat to some really revolutionary ideas in Google getting to the prototype stage before being squashed in the gears of "We're chasing a market and that's not how users see this product working." Stuff where, if it caught on, it'd be a paradigm shift... But it turns out users don't want every paradigm shift that comes down the lane.
Because Google has (traditionally; this has changed in recent years) a real push-pull in authority between management and engineering leadership, the company can't commit fully to building a quality implementation of a status quo. Nor can it commit fully to chasing entirely new ways of doing things that could shake up an established market. In general, this... Actually kind of works out fine for them, more fine than critics often realize, because neither of those answers are always correct. Sometimes you get Gmail. Sometimes you get Google Drive. And sometimes you get iGoogle or Wave. And sometimes you get the stuff in between, like Reader or App Engine (really popular among the users, but the users don't have the money to make it profitable to commit to it).
This is a very simplistic notion of what enables the creation of killer products. That has much more to do with understanding users' needs and identifying market opportunities. Good code writing processes are about code maintenance and scaling engineering effort, not dreaming up the next killer app.
I'll retreat to saying that claims about the superiority of Google's code writing process remain unsubstantiated.
During my 3 years at Google, I observed little brilliance. Not that my little corner of the giant org is a statistically valid sample.
I did see a fair amount of "Google is The Best!" sentiments and rejection of anything invented outside the company.
- The risk aversion to anything that threatens a big existing successful product.
- The product/feature approval processes that implements the above.
- The concern about launching something experimental or half-baked under the brand (vs a startup which does that by default).
Personally, I saw a lot of product and engineering creativity, but it was often stifled or watered down by the above.
It means that you wouldn't expect them to have products that fail because of unresolvable tech problems. You can see plenty of cases like Stadia where the tech was solid but the product strategy and leadership follow through was garbage.
Yet in the last decade, they really haven't had many successes (perhaps with the exception of Google Photos - 2015)
One would imagine that with nearly 200,000 employees at least one of them would have a good enough idea for a new product people like. But management and culture, with 'more wood behind fewer arrows', is no longer conducive to the good ideas getting launched.
I would argue that taking a product from 100m to 1bn users is a whole new level of success, and that has been done multiple times in the last decade.
Compute scales with the userbase. Engineering scales with the product featureset. That means a small userbase in general cannot support much engineering and therefore cannot have much complexity/features if it wants to be profitable.
That in turn puts big companies at a massive advantage. And in fact we see that - the vast majority of my time using the web is spent on big products (google, youtube, reddit, twitter), and only a small fraction on small sites (perhaps HN being the one exception here). Those big products won the battle for my attention.
And I'm not talking the UX, which has already changed substantially, but I mean the underpinnings.
There have been revolutions in apps. For example, pre 2017 I couldn't just tap a button and get an Uber. Or tap a button and have food delivered. Or do a video call with all my friends.
But the OS can't really claim credit for most of that innovation.
Not visibly. But the tech stack has changed substantially.
I'd find it quite an achievement if the UX has largely stayed the same while things under the hood got modernized over and over again and went with the times, subtly bringing innovations to the UX as well without people realizing. It's a feature that things don't look substantially different every two years.
https://appleinsider.com/articles/23/05/23/carplay-android-a...
> A report from Straits Research found that 98% of newly produced vehicles were compatible with either CarPlay or Android Auto. Meanwhile, 80% of prospective car buyers strongly preferred having these smartphone-based infotainment systems in their new vehicles.
The same research also shows Asia-Pacific to be the biggest market for these products, though North America is the fastest-growing.
Tbf. 25% connected is huge. If you just jump into your car to quickly do groceries or pick up your kids, you may not be interested in connecting your phone to your car, even if you really like that feature. So there could be a natural ceiling for the "connected" number and 25% feels getting close to that ceiling actually.
And everyone is just frustrated with laggy UI's in cars. But in reality they'll probably still just use Waze with a 10 buck phoneholder suction-cupped onto the windscreen.
People may have different preferences from yours.
Nevertheless, some people prefer not to accept them while not being entirely anti-tech either. You may not be agreeing with choices other people make, but it goes a little far to call them "absurd on its face", don't you think?
Not trying to downplay what happened after this: it was incremental change that made the products scale and be more usable to more people. I would say the things you describe as revolutions were actually evolutions of earlier products.
Which is somewhat offset by Google dropping Picasa.
I don't think this kind of analysis is really giving a good answer. I mean, by these standards, look what a disaster of a dinosaur legacy vendor Apple Computer is! Haven't had a successful product launch since the iPad 13 years ago! (Perhaps with the exception of Apple Watch - 2015).
TPU
Colab
Tensorflow
hmmm.
https://translatepress.com/is-google-translate-correct/
Another breakthrough is picture recognition. Did you know you can type in "motorcycles" and do an image search and there is ZERO text associated with many of those pictures of motorcycles? They really are mapreducing those pictures and running image recognition against a 17M open-source image library to assess "motorcycle-ness" of all those pictures! They started talking about picture recognition in the early 2000's but it really began happening in the late 2010's after they hired fei fei li, the mother of image recognition ...
Going neural definitely made a big difference but Translate was called "statml" by the infrastructure internally for years, maybe still is, because just using statistics and having training at all was a big deal back then.
Maybe it works well inside Google, but definitely not in the real world.
Arguably it was a bit thin on documentation but it just worked.
There is a tool that will do almost anything, but the tools are often just a bit janky. Rather than thinking of the toolchain as some polished, tightly integrated, perfect system, think of it more like an internal open-source ecosystem of things that work together but which are made by individuals, with limited resources, that still feature bugs.
This is not to downplay what we have, it's amazing in many ways, but most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them that cover almost every aspect of development. So much of the time I run into neat little systems and then discover they're a few Python scripts running under someone's personal quota. Sometimes these are 20% time projects, sometimes they are bits of tooling teams build for their own needs, sometimes they are more concerted efforts.
There is a core, tightly integrated set of systems that work well and are more "productized", but most of those aren't that much more special than what you can buy at any other company. If you've used Datadog then you'd feel at home in much of the modern infra tooling. Spanner is great, but anyone can pay to use it. Our CDN is very good, but there are other CDNs and from a developer perspective there isn't a whole lot of difference.
If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).
I would be curious as to how you view projects that are staffed entirely 20%.
That's true at Google, just as it's true anywhere. But what makes Google a much better experience than most places is a willingness (and ability) to pay for internal dev tooling of every sort with dedicated headcount. And that results in a consistently better experience than pretty much anywhere (I understand Meta has a comparable culture and ability to fund it, and I wouldn't be surprised if some or many of their internal tools surpass Google's). It's also unfortunately not replicable as a strategy, because you need a giant money spigot to make it work.
Why? Why reinvent the wheel 10 times?
But even when there's a similar open-source tool, it may be difficult to integrate with internal systems. There's a lot of benefit to a consistent set of technologies (data formats, storage systems, etc) so that's often a reason to write internally.
There is a lot of open source software in use at Google so I wouldn't say there's a lot of reinventing the wheel. And when there is an internal clone, the requirements are often necessarily different enough to warrant writing an internal version.
I found the same thing moving from a very large company to a smallish company. Both had good tooling, the latter just had significantly less and there were gaps in coverage. Whenever I encountered one of those areas I often found myself wishing I had access to some niche tool I had gotten used to at my prior company.
There are systems to resist this and people whose whole job is to focus on unifying systems, but they aren't perfect. In general, I'd say that Google is way way way more uniform than most other companies because of the monorepo, centralized tooling, and (mostly) uniform development process across the company.
But this only solves a visibility issue (and codesearch/moma is already pretty good at this). It doesn't help you when the tool supports a different language or the team that owns it isn't willing to commit to the SLO that you need or whatever.
My own experience at FAANG and non-FAANG is that __some engineers__ at FAANG get to dream big and demonstrate ingenuity, whereas at non-FAANG (especially startups I've been at) everyone can do this, though not all take the opportunity.
At my last FAANG where I was a manager, my opinion is that performance management and the focus on bullet points, metrics, and peer feedback for review time is a huge limiter for everyone else. I spent a lot of time creating cover for people on my team to actually try something new and innovative. More senior management constantly wanted to know why person X on my team was working on a new thing instead of trying to move some incremental metric somewhere else. Based on my experience during performance reviews for senior ICs, risk taking wasn't particularly encouraged at L6 (or even L7) and below as the cost of failure was substantial.
There is one underlying technology special sauce they refined to an art form, which is building a reliable system on incredibly unreliable components. This comes from their inception when the two guys building out the company knew how to write code but not build custom machines; their initial attempts at racks (hard drives and motherboards mounted to flexible plywood) are hilarious-looking by modern standards and broke down all the time, so they had to learn how to write software that was fault-tolerant at every link in the chain. That resulted in building an infrastructure that was extremely supportive of experimentation (if the system can survive half of it disappearing, it can survive a software bug crashing half the machines), and the rest of the shape of the company kind of flows from that.
It is worth repeating a lot that most companies do not operate at the scale or constraints Google does, and their approach to engineering is emphatically not one-size-fits-all.
Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.
One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations.
It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.
From an outsiders, it looks like all of FAANG has scaled to the point where delivery, perf and managing the business has become quite cumbersome. This, I suspect is why they're all used the economic climate to cut back, as they largely are unable to see the benefit of the additional hiring.
I have no doubt most of the talent is top tier, but being a strong engineer doesn't deliver profits nor cost savings on it's own.
More colloquially, this is phrased as “If you liked it, you should have put a CI test on it,” which we call “The Beyoncé Rule."13 From a scaling perspective, the Beyoncé Rule implies that complicated, one-off bespoke tests that aren’t triggered by our common CI system do not count.
Good CI is good because it easily integrates "one-off bespoke tests", not because it outlaws them.
And the reason for the disconnect is that the former centers the requirements of integration and not development, which is exactly backwards. CI isn't "the place where you run tests". The place where you run tests is underneath the software developers' fingers. CI is where you integrate tests developers are already running. And to the extent that is not true, CI stops being a useful tool for development and turns into just another roadblock to evade.
Only if your CI and developer workflow are different. To a first approximation, at Google, I run `blaze test //foo` to test `foo`, and CI invokes the same command when testing `foo`.
There's caveats here, but they don't apply to the beyonce rule in practice, which is centered on unit tests that can be run hermetically on presubmit. Your weird 3-hour integration test won't be run.
Edit: Actually it seems like you're saying "once you've tested a feature once, you can release it", which...no. You need some way of ensuring you don't regress, the CI suite is how you do that, otherwise no one will know when the new release of some random open source library that you depend on breaks your application.
Is there a version with links from one chapter to the next? Or a PDF?
and dont tell me I can override it in my browser. Neither my Android phone or Android Chrome honors my expressed wish to override everything with a high contrast, dark theme. A few honor it, many apps and docs do not. That article does not.