Hacker News Reader

Comment by BigJono | Hacker News Reader

It's always a lie? You're basically arguing that code with bad test coverage never works.

Production code is copping constant manual end-to-end tests. Those tests are more valuable than most unit testing, and some automated black box testing. I'll take stable production system over the unit tested new code any day of the week.

View on HN

Here's the an important piece of real-world statistics for you, which I've read about in research papers and personally verified in large-scale systems:

For every 1 complaint or help desk call you receive, there are between 100 and 1,000 users that experience the same issue but didn't bother to complain or call.

Production testing is literally 0.1% to 1% as effective as actual testing.

Depending on your user numbers, this works out pretty well. If you've got 100k users, you're likely to get reports on anything that hits at least 1% of them.

In my experience, you can write much simpler code that only needs to work in production than code that also needs to be unit testable. Adding hooks and separating bits and pieces so they can be tested can often make the code much less concise and harder to follow.

I would rather spend engineering time making sure problem reports from users get read and escalated and addressed than on testing things that are better left untested. Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect.

My favorite example of "things that are better left untested" so far is a method that returned a hard-coded array and a test which checked whether the method really did return that hard-coded array. So, if you wanted to add something to that array, you now had to add it in two places. Sure, that's an easy way to increase test coverage, but maybe you should focus on areas where unit testing makes more sense first?

I prefer to fix bugs before they hit production. It is almost impossible to do that without tests. Also testable code is usually much more readable than untestable god objects, tightly coupled modules and spagetti code.

without tests doesn't mean bad architecture.

I can go the other way and say that code that ends up being 'modified' to be testable, with 43 mocks to just be able to setup a class is as much crap as code without tests.

I would say I never achieved more than 20% or so of test coverage in my products and production code. It doesn't mean it is a mess of tightly coupled modules in any way.

But TDD and any other dogmas in development are like this. Solution A prevents (maybe) a problem, thus if you aren't using it all other code has that problem and defendants will just keep trying to make it so.

If you need 43 mocks to unit test a class, your TDD has already found a bad smell. Why are there so many classes tied to a single component?

Video games frequently succeed despite colossal bugginess. Pokémon Go’s launch is a great example. Engineers really undermine their authority when they talk about testing and quality in abstract and isolation to the user’s experience.

> Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect.

This is really missing the point of automated tests. Unit tests are for developers and ensure that low-level functionality doesn't break when changes happen. Integration tests verify that components fit together and that it stays that way. End-to-end-tests ensure that user functionality doesn't break.

Together, the above turn refactoring and rewriting of components from a business risk into an activity that can happen simultaneously when adding new features. This can ensure that there would never be such a thing as legacy code.

Legacy code is just code that has proven itself to be useful to the business. If you're telling me spending all day fiddling with testing is going to mean I have nothing to show for it, I guess I agree.

I think there is room for testing, but you need to really restrict the scope to those parts of the code where the requirements don't change often, and where the natural interface is amiable to testing, and often, you need someone whose job is testing and who didn't write the code to actually do it. Incidentally, those parts of the code where the requirements don't change often get to a point where the code doesn't change either, which should mean the tests won't fail, because the inputs stayed the same.

  between 100 and 1,000 users that experience the same issue but didn't bother to complain or call

Which is often because users have no (established, straightforward) process for reporting, or recognizing, a defect.

I found an ugly 2FA bug in a major ride-sharing app's onboarding and was shocked to find that they have no informal mechanism for reporting any defect. (I thought it important enough that I went through the hassle of reporting it through HackerOne, not seeking compensation but simply attention to the problem. It was closed within minutes without a glance or meaningful comment.[0]

I'm not shocked anymore, as this seems to be increasingly common.

Another growing trend is meaningless and even juvenile error messages. "Oops!" (with no context that could aid diagnosis). "Something went wrong." (ditto, and I see this out of the leading ticket site 300 times a year). Etc.

[0] it's possible that it was closed because that particular onboarding was later obsoleted, but still, that's just a bad look.

That's the kind of thinking that leads to situations like this:

https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultr...

A lot of the time it's just as important to make sure certain things can't work as making sure that certain other things will.

> It's always a lie? You're basically arguing that code with bad test coverage never works.

It may work, or it may not work. Without the tests you won’t ever actually know.

Maybe your customers are bouncing on bugs, and while you are earning enough, you could have been earning 3 times as much without those bounces.

Reply on news.ycombinator.com