Hacker News Reader

Comment by simonw | Hacker News Reader

"How much of a burden could this really be?"

My understanding from conversations I've seen about Google Reader is that the problem with Google is that every few years they have a new wave of infrastructure, which necessitates upgrading a bunch of things about all of their products.

I guess that might be things like some new version of BigTable or whatever coming along, so you need to migrate everything from the previous versions.

If a product has an active team maintaining it they can handle the upgrade. If a product has no team assigned there's nobody to do that work.

View on HN

My understanding is that (at least at one point) binaries older than about six months were not allowed to run in production. But APIs are "evolving" irregularly so the longer you go between builds the more likely something is going to break. You really need a continuous build going to stay on top of it.

Best analogy I can think of is log-rolling (as in the lumberjack competition).

Google is famously a monorepo and is basically the gold standard of CI/CD.

What does happen is APIs are constantly upgraded and rewritten and deprecated. Eventually projects using the deprecated APIs need to be upgraded or dropped. I don't really understand why developers LOVE to deprecate shit that has users but it's a fact of life.

Second hand info about Google only so take it with a grain of salt.

Simple: you don't get promoted for maintaining legacy stuff. You do get promoted for providing something new that people adopt.

As such, developing a new API gets more brownie points than rebuilding a service that does a better job of providing an existing API.

To be more charitable, having learned lessons from an existing API, a new one might incorporate those lessons learned and be able to do a better job serving various needs. At some point, it stops making sense to support older versions of an API as multiple versions with multiple sets of documentation can be really confusing.

I'm personally cynical enough to believe more in the less charitable version, but it's not impossible.

I agree this is an overriding incentive that hurts customers & companies. I don't think there's an easy fix. Designing & creating new products require more relevant capabilities from employees for promotions then maintaining legacy code.

> I guess that might be things like some new version of BigTable or whatever coming along, so you need to migrate everything from the previous versions.

Arrival of new does not neccessitate migration.

Only departure of old does.

But if you don't downgrade the old, then you're endlessly supporting systems, forever. At some point, it does become cheaper to migrate everything to the new.

They deprecate internal infrastructure stuff zealously and tell teams they need to be off of such and such by this date.

But it's worse than that because they'll bring up whole new datacenters without ever bringing the deprecated service up, and they also retire datacenters with some regularity. So if you run a service that depends on deprecated services you could quickly find yourself in a situation where you have to migrate to maintain N+2 redundancy but there's hardly any datacenter with capacity available in the deprecated service you depend on.

Also, how many man years of engineering do you want to spend on keeping goo.gl running. If you were an engineer would you want to be assigned this project? What are you going to put in your perf packet? "Spent 6 months of my time and also bothered engineers in other teams to keep this service that makes us no money running"?

Because it costs money to run things, and no one wants to pay for something that they aren't getting career value for.

> If you were an engineer would you want to be assigned this project?

If you're high flying, trying to be the next Urs or Jeff Dean or Ian Goodfellow, you wouldn't, but I'm sure there's are many thousands of people who are able to do the job that would just love to work for Google and collect a paycheck on a $150k/yr job and do that for the rest of their lives.

I'd like to encourage you consider the following two perspectives --

1. A senior Google leader telling the shareholders "we've asked 1% of our engineers, that's 270 people, costing $80M/year, to work on services that produce no revenue whatsoever." I don't think it would pass that well.

2. A Google middle manager trying to figure out if an engineer working exclusively on non-revenue projects is actually being useful or otherwise; this is made more complex by about 30% of the workforce trying to go for the rest and vest option provided by these projects.

> A senior Google leader telling the shareholders "we've asked 1% of our engineers, that's 270 people, costing $80M/year, to work on services that produce no revenue whatsoever." I don't think it would pass that well.

The business case for this is that Google lose a bunch of money in b2b (cloud mostly, potentially AI in future) because professional users (developers etc) don't believe that products will be supported. Every time Google shut down a service like this, this perception is re-inforced. We're investing this money into these services to change our brand perception and help us make more money in future.

As a bonus, this kind of cultural change would also force them to rebuild their engineering systems (and promotional systems) to make this easier. This may not have mattered for Search/Ads but it will matter if they actually care about winning in cloud and AI.

A Google shareholder that shortsighted might as well ask why they have an HR department or have custodians to maintain the offices, after all, they don't generate income either.

The manager in the trenches can tell if there's actual work happening, to move goo.gl from the internal legacy system to the new supported one doesn't magically happen, code needs to change for it to work after the old system gets shut off.

A lot of Google infra services are built around the understanding that clients will be re-built to pick up library changes pretty often, and that you can make breaking API changes from time to time (with lots of notice).

And you could assign somebody to do that work, but who wants to be employed as the maintainer of a dead product? It’s a career dead-end.

> If a product has no team assigned there's nobody to do that work.

This seems like a good eval case for autonomous coding agents.

Reply on news.ycombinator.com