Hacker News Reader

Comment by veltas | Hacker News Reader

A reason not mentioned is that testing public methods only is more black box testing. It's important to do this not just to allow significant changes to the implementation without affecting the test, but also because if you tie testing too much to your implementation you may miss holes in your interpretation of the requirements/interface that are better caught in a black box test.

If you want coverage then private method testing is giving you fake or inferior coverage. Those hard to reach paths are only truly tested via the public interface (i.e. called by the function that will call them in practice, not your contrived test function).

View on HN

This is so important but even most professional software developers and testers don't seem to understand it. Tests are not supposed to break on every little change to the code, or even sweeping changes in implementation. If they do, they're not very good test.

https://testing.googleblog.com/2013/08/testing-on-toilet-tes...

https://medium.com/expedia-group-tech/test-your-tests-f4e361...

I disagree. You are basically saying that only end to end tests are very good tests.

There are different reasons to have tests.

Unit tests often break even during small refactorings but are great to quickly cover of lot of edge cases of some complex piece of code. They also quickly point to what exactly is wrong.

End to end tests on the other hand should be almost immune to refactoring problems, but they are very slow and often it is tricky to find where the bug is or even if there is a bug at all in the application - in those tests bugs are very often in test code.

As always, all the extremes are wrong.

There are plenty of non-end-to-end interfaces that are well defined so tests don't break on changes and still encapsulate complex and important code. There is somebody on this thread talking about one that calculates the costs on a contract.

Tests for those are unit tests, and fit the GP's description. And those are the best units to test, often you don't want to unit test anything else.

Of course, YMMV. For example, on some languages you must test if the data has the correct structure and the functions have the correct interface, what is best done by unit tests, and those tests will break often. Some problems are so complex that you must test intermediate steps, otherwise you will never get it right; those tests also break often, but when you need them, you need them.

If refactoring is causing you to struggle to fix the unit tests, then what you have really aren’t unit tests.

Unit tests should be cheap to write and cheap to reimplement. Otherwise GP’s right and the tests end up shackling you to an implementation, instead of defining the requirements programmatically.

Just stop trying to fix unit tests. All of this noise about test bloat and the pain of refactorings goes away if you make one mindset change: feel free to delete tests.

If you make a change and a unit test breaks, do your best to make sure it’s not an actual defect. If it isn’t and fixing the test looks like more work than you want, then delete the test. Maybe make a new one to cover a similar case if that makes sense.

“Less tests” is a really easy state to get to. Just highlight and delete. If I think having more tests is good and you think that it’s bad because you don’t want the tests there if they break, that’s not a problem! We can get back to the way you want things in less than a minute.

Someone writing a test is not a sworn oath to preserve it forever. It’s not even like a public method. No one is using it.

There's an inflection point with unit tests where a good description, a good matcher, and a small number of preconditions (mostly handled by nested test suites) make it easier to rewrite the test than to fix it. When I say 'cheap' tests, these are what I mean. Tests you can treat like livestock instead of pets.

The sentiment of the test may still apply, but the implementation details mean you're testing, for instance, for empty set using a different strategy. Block delete and start over.

Or the feature you're adding is meant to allow us to deal with empty set: let our customers put things in their shopping cart even though we don't have any shipping addresses for them. In that case we 'broke' that requirement, and just delete the test and move on.

But getting cheap tests and high coverage is a bit of an art, often learned via the school of hard knocks. If you let too many requirements leak up into integration tests, you may find you have many, many tests to fix when a "never" or "always" changes to "occasionally", "mostly". Some people get brave and delete these tests, and discover Chesterton's Fence later when someone yells at them for deleting regression tests, because the test described 2 scenarios it was asserting, but it surreptitiously asserted 2 more, one of which the New Guy just broke but got a green build.

With a unit test it's tenable to make a test that looks like it checks one rule while actually checking only one rule.

See, I think your example of breaking the thing is where things are tricky. I know you’re not advocating for it, but I feel like a lot of people get burned by an experience like that. They think “Someone wrote a test. Then it broke. Then someone deleted it or fixed it wrong. Then something else broke and no one noticed because the test was deleted. Therefore, don’t write too many tests.”

Isn’t that last step super weird? I don’t see how not writing tests would have prevented the regression. At most vague hand wavey “people would be more careful” which I don’t buy. The problem was breaking a dependency. Tests are not themselves dependencies, they are an automated stand-in for them.

What if you have a "module" within your private code?

An example I have is in a pay computation engine. I have a public function that computes how much someone ought to get paid for a given shift based on who was working the shift, what they were doing and a bunch of other criteria, what day they were working on, what time of day, etc.

Within that I have a private function that works a with a narrower scope that just does the pro-rata calculation for time-of-day and day-of-week. That function itself has a fairly high-level interface with a highly complex implementation. IMO it makes sense to test that function, even though that function is private and is only used by the one public function.

My point being whether it's private or public isn't really the point. The key thing is whether the functionality is self-contained and decoupled from the rest of the code with a stable interface.

The 'module' is a library of pay computation functions that your app uses privately.

The library itself is public and could have other users, eg. you when invoking manually from cli when testing things out.

Test your library code with unit tests - this is easier because it can be done in isolation feom external systems.

Itegration testing of apps is difficult. Often it is sufficient to test by use - have a dev environment which is usable or test in prod if it's not a critically important app.

This is often sufficient, but I'm sure testing gurus have a recipe to apply unit tests to the app itself as well. I personally prefer not to have 10:1 test to code ratio, so I keep things practical.

Based on my experiences, I don't think anything should be truly private. I found it's almost always better to put the "private" code in a separate "implementation detail" namespace that makes it clear it's intended to be private.

This hides the private implementation details, but keeps them accessible for when it's needed for internal tests or workarounds. It's easy to find code which relies on the private namespace, and one can't accidentally use it.

In Java I would sometimes make them protected methods and use a subclass at the top of the suite to expose them.

These tests that cross from black to white box testing could be construed as functional tests, in which case my solution and yours aren’t so different.

Also important to remember that it’s a mistake to treat library and application code as the same thing. They are not the same, and from my observations a lot of arguments about The Right Way come down to trying to treat them as if they are.

Indeed protected methods can be useful for this. Depending on the task at hand, I find it can be useful to separate code into it's own namespace (one way or another) to make it more clear what the user-facing code does.

Regardless, as you noted, my point is to not prevent access but rather make it obvious if you're using implementation details.

Most test frameworks are very un-opinionated, and perhaps that is a mistake.

Framework doesn't care if half your unit tests are functional tests, if you slide E2E test code into the before*() call in an integration test. If your white box and black box tests are snuggled up to each other in the same test suite. If your tests only pass if they run sequentially, in this particular order.

It's no wonder we can't get the terminology straight. You can be years into writing tests before anyone or anything calls you out.

> Also important to remember that it’s a mistake to treat library and application code as the same thing.

They have the same pattern at two different levels. What’s the essence of public? It’s that there’s an encouragement for people to use it and you don’t know who is using it. Private is the opposite. Its usage is discouraged or not allowed beyond a certain boundary, so you know everyone using it in an endorsed way.

In a library, your public methods are public and your private methods are private. In a service, all of your code is private! Critically, it meets the definition that you can see all of the usage.

So, why do we test public methods in services? (I’m sure someone would say not to). We just want to be more confident they work as expected now and will continue to do so in the future (I think people forget that’s a [the, even] big reason for tests). Sometimes testing something “more private” gives you faster, more accurate tests.

Maybe you can see it coming that I’ll argue by analogy it can also make sense to test private methods in libraries. They’re not part of your external API. But, just like public methods in a service are an internal API, private methods are part of an internal API for the class that is used by your future self and coauthors.

> private methods are part of an internal API for the class that is used by your future self and coauthors

Indeed. In a good code, one could say it's libraries all the way down.

For example, I've refactored a lot of business code such that it can be easily re-used in different parts of our application. Ta-da, I effectively have created a piece of library code, and I'll want to write tests for it to make sure it doesn't break any consumers, current or future.

I find testing everything through public methods can lead to excessive mocking, which ties you even more to the implementation.

How does mocking tie you to the implementation?

> coverage then private method testing is giving you fake or inferior coverage

Absolutely not

Your private methods contain logic no?

Every coverage is worth

In fact I don't see any "controversy" here. Not testing private methods seems like a lousy excuse

exactly because they change less with implementation/api changes and are more "logic focused"

Sure, do test your API methods, but that might not always be easy or practical

I think you misunderstand. GP is recommending that coverage of private methods be achieved by testing their public consumers. If you can't test part of a private method through a public consumer, then that's a sign that those unused code branches might not be necessary.

Testing public methods is not an argument for not testing private methods. In testing, unlike in construction of software, it can be good to combine many different approaches, because each has diminishing returns and they compensate for each others weak points.

Reply on news.ycombinator.com