Hacker News Reader

Comment by bluedino | Hacker News Reader

Most places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness.

The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'.

The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some change was made, usually in a hurry to meet a deadline, and it's been broken ever since. This one is usually a result of the fix being too difficult, because the test environment was haphazardly thrown together in the first place.

View on HN

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'.

One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similarity to prod to be useful.

As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is often so complicated (banks, insurances...) that it should have been considered early during design.

So, I wouldn't use the word "lazy", but we could do better.

It's not the architecture, it's the representing all the states in prod in testing.

Not saying it's impossible, just not always feasible to have it actively "perfect".

I've found that an "ok" testing envrionment and solid unit testing / monitoring tends to do better.

And then b/g deploys and canaries :/

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.

> Modern architectures look like this:

A lot of modern architecture is not like that.

And even the diagram there is something should be possible to replicate if the organization values it.

Today it doesn't even need to be too expensive as it can be deployed with terraform and torn down an hour later when the full system tests are finished qnd go back to running the integration tests.

It's not possible to replicate the CDN, or DNS, or the message queue if it's hosted, or the database of it's big enough, or etc. etc. The differences between what you're able to create in a staging environment and what exists in prod are significant enough that testing against the staging environment doesn't build much more confidence than testing with mocks.

Testing in prod means accepting this reality, avoiding the largely unproductive toil of building and maintaining a staging env, and making the production system observable, resilient, and operationally agile enough that you can deploy changes that can trigger unpredictable and emergent system behaviors while managing risk appropriately.

To be useful? That seems to be an extraordinary low bar to hurdle for a QA environment. I imagine most clear that bar easily. I know mine does.

Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?

I'm not the person you're responding to, but having the same software and hardware is possible (most of the time), but production environment is a living thing. To simulate production you need the same data and interaction you have in production, and that's quite difficult to do in many systems, if only for regulatory reasons (PII, PCI and friends).

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

All of the pieces have complex and emergent behaviors as they interact with each other. Many (most?) are hosted, not even fully in control of the team using them. Recreating this environment in a hermetic way for testing is either actually impossible, or so cost prohibitive as to be practically infeasible.

Not sure where you are working but not a lot of people work at/on a huge code base.

Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

The size of the codebase is orthogonal to the size/complexity of the production environment.

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.

I know this very well and i'm stlil not seeing an issue.

We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.

Same network rules and configuration? Same CDN settings? Same DNS? Same data in the databases? Same volume of traffic on the queues?

(Rhetorical questions. Of course the answer is no.)

All of these things have huge impact on the actual similarity of the systems, and consequently how they behave.

Obviously but this is not a black and white thing.

If it wouldn't make sense for us to have test and prod, we wouldn't do it but i can't imagine a scenario where this is NOT beneficial.

Reply on news.ycombinator.com