Hacker News Reader

My intuition, and some experience sitting in meetings with GCP folks, is that their engineering teams don't dogfood their own products end-to-end sufficiently (e.g. including billing) on a daily basis, like their customers have to.

The amount of blank stares and "Oh..."s that happened when asked about relatively simple, everyone-would-need-it use cases for management, visibility, etc was mind boggling.

GCP feels like Google rediscovering being Microsoft of the 1990s. If you have strong product teams, but no strong overarching experience teams, your resulting system is going to be a hash of well-polished but distinct products, with an extremely ugly unification layer.

I was reminded of IBM trying to get them to show up and actually sell things. It is _bizarre_ that you have to hound sales people to actually make an effort — it really seemed like they assumed the Google brand was enough to guarantee buyers and were surprised that anyone would question whether their products were the best.

(This was also the first time I heard Reader mentioned at the C level as in “what will we do when you cancel it?”)

> (This was also the first time I heard Reader mentioned at the C level as in “what will we do when you cancel it?”)

Everybody who loved reader is now at the Director/VP/Csuite Tier, the sunsetting of reader also burned so much goodwill

Well, not everyone but it tended towards influential groups — they burned so many tech journalists that it really seemed to usher in an era where goodwill was no longer assumed.

I really do think this dogfooding is why AWS is successful. Amazon's businesses run a lot of their workloads on AWS. Amazon is AWS's largest customer. So AWS has the benefit of having thousands of heavy use customers internally to discover bugs and edge cases and provide feedback.

For example, I contributed a fix to AWS documentation as a SDE in the Kindle org. This is the kind of improvements you get with dogfooding.

You can tell which parts Amazon dogfoods!

Billing is terrible, although it has gotten a bit better. Cognito is probably one of the worst services on AWS, and it's only getting worse (there are now two SDKs with different APIs for no reason at all). While things like EC2 and Lambda work pretty well.

You can tell which parts Amazon dogfoods! Billing is terrible, although it has gotten a bit better.

I don't have much detail on Amazon's policy, but there was an AWS devrel on Twitter a while back saying they had to run and pay for their own AWS account as if they were any regular user for their own playing around/research/etc.

The AWS billing system probably grows in complexity in response to internal Amazon politics in that case, unless someone up top stopped that.

> You can tell which parts Amazon dogfoods!

Same for Google. Everything that's in use by Googlers is pretty dope - calendar, video conferencing, docs, search obviously, maps, etc. Everything that's not - less so.

But.. what about Search? Maybe they don't need search from internet but need search from internal?

Search was on my list, as a former Googler I used both the external and internal search all the time.

You may well be right about that. I can't imagine that if you have the power to fix it that you would accept it the way it is.

It reflects the company culture. Unlike Amazon, where customers experience is their top value, at BigG they seem to build stuff for the sole gratification and ultimately promotions of engineers and managers. They don't seem to care much for their customers.

It's easy to dogpile on BigCo, but this is selling the employees and teams short.

They definitely care about their customers, and many things were made better through subsequent fixes.

But the larger point is that the processes and mid-high+ level management structure at Google don't seem to prioritize cohesive, customer-centric experience. Which means teams will always miss things... because the process doesn't ensure they're caught.

Hardly a surprise. From what i have heard, it sounds like Google's internal use of GCP is mostly automated, using internal apis. I.e. it sounds like are not clicking through the screens, or even using the external version of the APIs. I question if the externally available screens are even capable of handling some of the special permissions that google internal workloads can be given.

This is in contrast to say Azure, where plenty of Microsoft employees are using the same resource manager APIs and even using the portal. I think even the billing related features get used as part of internal budgeting (they want teams to try to keep resource utilization reasonable). While teams developing parts of azure itself may be utilizing internal APIs (for example Microsoft Graph is basically just a giant wrapper around a variety of internal APIs), most of the rest of the company sees and interacts with azure in the same way we do. (Except that they also have access to dogfood/PPE environments that we don't, such that endpoints for say integration tests don't need to run on production azure).

It's true enough, but it never really felt like teams solicited feedback on their APIs or portal UX. Azure really only made this jump to using all public products internally recently; even the CI systems used by internal teams were proprietary until recent pushes to move to an ADO-centric model.

I'm also not certain that Azure really has the right internal pressures to produce great UX results. In my experience Azure's culture internally is very lackadaisical with only a few teams really pushing the platform forward.

As a power GCP user, I see them running UX studies all the time. Having been using GCP for the last decade, the UX updates feel like day and night changes. Frankly I find the AWS console a lot creekier.

> This is in contrast to say Azure, where plenty of Microsoft employees are using the same resource manager APIs and even using the portal. [...]

And still they don't have this incredible complicated and not understandable IAM. Most user I know just give everyone root because it's not possible to just allow some specific API operations for a specific set of credentials. Or maybe I am to AWS.

> I think even the billing related features get used as part of internal budgeting

Not good enough when I kept being charged on an account I closed.

A fair amount of my own confusion with GCP's offerings comes from their decision not to use proper names for their services.

AWS may have arbitrary names that don't follow any patterns, and Azure may have names that are grandiose, but at least you know with both of those clouds that they will always capitalize the name of all their service/product in documentation. There's no confusion if they are talking about a load balancer in the abstract, or their specific managed offerings.

Google does call their own balancer GCLB though.

Is there a single thing called Google Cloud Load Balancer? My understanding is that there are about half a dozen different managed load balancing services that all have very different operating characteristics. They all have very generic names in the docs like "global external HTTP(S) load balancer."

Much the opposite. GCP's names are painfully transparent and banal. No need to look them up, you can tell what they do from their names (which are proper nouns, so it's clear they're a service name)

- Cloud Logging - stores logs

- Google Compute Engine - VMs, compute

- Cloud Monitoring - stores metrics, monitors stuff

- Cloud Storage - stores stuff

- Google Kubernetes Engine - runs Kubernetes clusters

- BigQuery - runs multi-billion row queries with ease

- Cloud Functions - FaaS run time

Now sure, Cloud Spanner is a DB, without DB in the name, so you'll have to read the docs for a few seconds, but let's compare that to Cognito (brains?), X-ray (medical imaging?), Kinesis (something to do with mitosis?).

As a more generic term, kinesis just means "movement"/"motion", in Greek,

My own frustration is not with the overall product line names. My frustration is when specific options of each product line are discussed in relation to each together in documentation.

Take the example of "GKE Ingress for HTTP(S) Load Balancing." It's overly generic and doesn't tell you at a quick glance whether you are buying a specific managed load balancer or running something yourself, and whether you are using basic exits to the public internet or buying into a global network accelerator.

I don't know if I am the only one but GCP console and tools is the easiest and most logical to use for me compared to Amazon and Azure.

Same here. GCP is far more logical and consistent compared to the others. There's just a level of complexity at some point that you have to deal with or use a simpler provider.

Seriously. I don't know what GCP these guys are using. Compared to AWS, using the cloud console is like ordering a pizza online.

As an owner on the account, I kept getting an error that I didn't have permissions to see the billing pages.

I contacted support and the first thing they asked is which browser I'm using. Brave.

Turned off the shield and everything magically worked. I got a small laugh out of that one.

I ran into this in chrome. The org entity had billing, but I was project admin but not billing, but could give myself billing permission?

Shutdown project also needed some billing permissions maybe? Cryptic messages.

I actually like the "project" permissions model, but I don't like how often I have to figure out why clicking around in the admin page as the "admin" causes problems.

On AWS side the admin policy is pretty broad, and the root account seems to really work as root.

Admin policy (AWS):

        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }

In AWS they have organizations where root does not always work as root.

> how it is perfectly possible to orphan an entire project[1] if a person leaves your org

Maybe not the best example since this (unlike other IAM oddities) actually makes sense - it can only happen when you don't have a top-level org tied to a project, like when you do something like using a gmail.com account to spin up GCP resources. Inside a GSuite org, this is not the default and I can't imagine how it'd happen by accident.

If your project is not attached to an org, and all the accounts tied to are gone, then what else do you expect?

> Gsuite and GCP may superficially appear to share a bunch of stuff but that just sets you up for some very cute surprises, from which it can be extremely difficult to recover

The way it's implemented is actually quite nice for complex scenarios/defense in depth - for instance, you can set it up such that whoever owns the GSuite org does not automatically get access to all GCP resources. Of course, any security measures good enough to restrict an org admin's privileges also have the potential of locking yourself out in a way that's semi-irrecoverable.

I can't believe how confusing all these cloud products are to do the most basic things. It really makes me appreciate Cloudflare, they seem to do a really great job with their UIs.

> Google cloud has to be the most confusing product suite known to mankind. What an unbelievable mess.

No, that's AWS.

I'd say Azure is the worst. It's like Windows control panel with 10 times more items except you don't have the muscle memory from older Windows to navigate it through.

And then some edge case where I was charged on an account I closed because some subscription was still left open while I could no longer login...

It would be nice if most of Google ran off GCP, but alas, few marquee Google properties run on GCP. Ever notice how Google search, GMail, etc are usually just fine during a GCP outage?

Well this is just untrue. Read the docs. Gcp uses lots of the same infra as other Google services. The whole global load balancer infra is one and the same. Spanner is used by maps, Gmail etc. In fact, that's where the service spawned from.