Hacker News Reader

Comment by lexicality | Hacker News Reader

> The problem with this approach is it’s completely tautological. What are you asserting? The tests themselves.

Every time I've heard this kind of rationality it's from someone who doesn't understand what a unit test is and is pushing for integration tests.

It's extremely useful to have very rapid tests that boil down to making sure `add(2, 2) == 4` because you can have them running constantly and catch inadvertent logic changes.

Should you also have more heavy tests using local/fake services? Obviously - but it's not one or the other.

View on HN

I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I have a weakly-held preference (that I will not fight for in code reviews) for not unit testing imperative code at all because I think it has very limited usefulness and is just more code to maintain. Instead, I prefer to rely on integration or end-to-end testing to validate the behavior of those parts of the system.

But I think unit tests are extremely useful for functions. It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. I think it is always useful to have edge case tests - when is it an error, what happens right on the boundary between error and non-error, etc. - and to have at least one illustrative happy-path example. (I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification.)

Where this breaks down is when the vast majority of code is imperative, which is very common in practice. My usual approach there is to lean heavily on higher-level testing, and refactor out little chunks that can be more reasonably unit tested, when practical. And I also write and maintain lots of these not-very-useful imperative tests, because people get itchy about low unit test coverage and I'd rather write the tests than spend time convincing people that they aren't very useful.

> I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification.

I am that person! I am a big fan of property based testing and try to add such tests whenever possible. They serve as "edge-case hunters" and are usually part of my programming loop. As soon as something has been modified, a test suite executes. Writing property tests first and the code later allows you think about the invariants and behaviour of what you are building in a way that regular unit tests just don't.

Yeah I get it. But if I have a function where the domain is signed 64-bit integers and the specification is some f(x) between zero and a billion and an error otherwise, I don't want to iterate through all the legal inputs and make sure they do the right thing. If I'm using a language that can do that with its type system or other kind of capability, then I'm happy to use that! But I'm also not grumpy about using python or rust or all the other normal languages just because they can't do this. I'm happy to just write a few cases mostly concentrated around zero and a billion.

Technically you can do that with property based testing just fine, e.g. for all values outside boundaries ensure a throw. The goal of property based testing is to help you identify invariants in your code and then verify them.

I just looked up Hypothesis in python and it does indeed look pretty cool for this kind of thing. I should try this more!

The best part imho is the shrinking.

Clean functions with limited dependencies which encode non-trivial domain logic are indeed well served by unit tests - even if theyre a bit imperative.

But, outside of contrived interview scenarios this type of code is actually somewhat niche and outside of that niche unit tests suck.

Ive worked on tons of projects where it accounted for 1-2% of the code base. Unit tests not only werent capable of catching bugs they werent even capable of replicating 98% of them. Unit test theater.

Meanwhile, the other day I did TDD with hermetic end to end tests using playwright, mitmproxy, mailpit and screenshot/snapshot testing. It felt like I was living in the future.

In most software, the units themselves are not that interesting. Basic business logic and data transforms, surrounded by a larger codebase that glues all the integration points together into a working stateful system.

I've met a few unit-testing extremists who insisted on pristine-mocked unit tests and only unit tests. Their tests were incomprehensible (a good test should serve as documentation and acceptance criteria - but their code was 90% of concerned with mocking service apis) and importantly, their unit tests repeatedly failed to catch serious regressions that should have been apparent to anyone who simply executed "main" without mocks. We had production outages on the regular.

The funny thing is, the unit test extremists doubled down, rejecting the idea of integration or e2e testing entirely (yes, I'm serious) and claiming that prod failures indicate we need to unit testing EVERYTHING. They even went so far as to claim developers need not actually run their software, unit testing is fully sufficient. Not surprisingly, their code had perfectly constructed units but no overall design... the units composed into an unmaintainable dumpster fire as an actual system.

I would generally argue that if your unit tests have mocks, they're not actually unit tests. What you're describing sounds like integration tests that completely missed the point (i.e. the "integration" but).

Even very small "units" have dependencies in the sense of other objects or functions that they use. Or at least they should, if concerns are being separated. It's good to use fakes or mocks to make those dependencies not depend on state external to the test, but not to structure tests around what methods are called on which dependencies in what order.

Yeah, I guess I would make the distinction between simple stubs and heavyweight mocks and fakes. If you're using a lot of the latter in your unit tests, that's a code smell to me; unit tests should ideally not be concerned with the state of the world, which is where mocks (especially fakes) tend to come in. Obviously there are no absolutes, but that's my rule of thumb and it's served me well.

I agree. Mocks tend to be complicated when the api of the dependencies they're mocking is complicated.

What you call them isn't really important. "Unit tests with mocks" or whatever they are still suck.

I think the distinction is still important because it frames how you should write the test. Unit tests basically just assert that, given some inputs, the output of the "unit" is what you would expect. Integration tests, on the other hand, test the business logic that ties different units of code together, which changes the assertions to be less about the direct output and more about the state of the world. For example, if you have a Twilio wrapper, the tests for that wrapper are unit tests because you're just testing that the wrapper makes the correct HTTP calls, etc; but the code using that wrapper should be tested via integration tests to ensure that the correct calls are made, that state transitions are handled properly, etc.

This might sound like a meaningless distinction, but I've found that a large portion of bad tests that I've seen in practice were integration tests written in the style of unit tests (i.e. treating stateful code as if it were just a pure function of its inputs).

Given that integration testing is inherently harder (as you've alluded to), I'd rather see more unit tests, but that requires an architecture that breaks code down into as small and focused of units as possible (e.g. a functional style), which isn't always feasible for practical reasons.

Well put.

Yes, this is an insane way to try to build software. I don't think it's very common at all, but I also don't think it is completely non existent.

Writing unit tests can help catch early bugs, but once they're done and green, I don't think anyone expects them to catch new and exotic bugs (it happens but it's so rare).

Instead they stay as a live and enforced documentation of how the target function behaves.

.. and behaviour validation for any rewrites.

I agree with everything else you said but I totally disagree that functions like that are "niche". It's true that many codebases are written such that they have few of them, but that's "accidental" not incidental. Almost nothing is pure math, but almost everything is data transformation that is deterministic given some set of parameters and dependency configuration. And mocking out external state to control the behavior of the thing being tested is great. But the point is not to say "and then this method of that dependency is called with this input", it's to control the behavior of the thing being tested, by controlling what its dependencies return.

> I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological.

I agree. I usually see the I/O layer weaved with the data transform layer. Often in the same function call. And, usually, the answer is to pull those two layers apart, put tests around the transforms, and push the I/O to the edges. Usually...

> It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic.

(de)serialization functions are a good general case purely-functional example. Data is transformed and you need to verify the transform was correctly performed, has all kinds of unique edge-cases, and should gracefully inform the caller when something is wrong.

Nothing to add, just: yep, all of this.

I’ve found those “obvious” checks to be useful to catch unexpected hardware issues as well. I had a customer upset that my code was crashing- I sent a debug version with asserts around some arithmetic core to the algorithm and lo and behold those asserts failed

His machines memory was bad and corrupting the internal state of the application. Until I could prove that his machine was calculating the equivalent of 2+2=5, he wouldn’t believe us.

he was talking about mocking, not tests such as CHECK(add(2, 2) == 4).

and BTW i would challenge anyone working with accessing complex real-world systems (for example, an stock-exchange, or a robotic warehouse, to write a worthwhile mock for them.

If I have a function that hits a JSON API and returns a path from the resultant object and I mock that API to return 4, I'm doing exactly that.

I can write a mock for every data type the API could return and one for every error the API can return and test all of them near-instantly without having to do a HTTP round trip to the fake server (which presumably also now needs configuring before each test)

i think you either over-estimate the complexity of your json api call, or under-estimate how complex accessing a stock exchange is

anything is mockable if your code is broken into smaller easy to reason about easy to test chunks and you have a complete understanding of what the thing you're mocking is able to do.

If you don't have a complete understanding of what it's able to do then I think how you write your tests is the least of your worries.

> anything is mockable

true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange (and mocked other clients than your client app). and then you are going to have to test that mock. this could all take a while.

i really don't think that people who suggest mocking realise how complex applications work and how they are always changing their interfaces - for SEs due to regulatory pressures, for example.

> true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange

All your mock has to do is to provide the specific answer you're expecting to receive from that service in the specific case you're testing. In most situations, re-imlementing logic in a mock is a mistake that should be avoided.

with a stock exchange, it's a dialog - don't expect any particular response. for example - using sell and buy just to keep things simple, and i am the seller and i could not be bothered looking up real prices or tickers:

sell 10000 BT at 50GBP // i put this on the exchange

buy 500 // somebody bought some

buy 2000 // somebody else (or maybe the same) bought some

crash // exchange crashed (not rare) we recover

cancel 500 // they can do this in a limited window

buy 8000 // we have filled

you see, it isn't simple, but that is about as simple as you get

Wouldn't each of these be a test case?

only if you have a stateful mock server with multiple mock clients attached to it. for example, when you put a sell on the server, you are basically going to be listening on a socket for multiple responses and what the responses are will depend on the buyers (which you must also mock). and don't get me started on buying, ticker conversions between different exchanges, currency, time, etc. it is too hard to mock.

to mitigate this some exchanges, such as london and paris, do provide test exchanges, which are crap, and available every other fridays, or whatever. when i worked in this area, my boss used to test my code by putting a trade on the exchange and then immediatly cancelling it. this was illegal, as he wasn't a licensed trader, and the exchanges got enraged. i refused to have anything to do with it.

but my point is - you cannot do meaningful unit tests on very complex systems. even integration tests can be very hard, but are worthwhile.

> but my point is - you cannot do meaningful unit tests on very complex systems.

I think you're missing the point: unit tests should not be testing entire systems; that's the point of integration or end-to-end tests, which are different beasts entirely. You can very much write unit tests against parts of a complex system because each test should only be testing one isolated bit of functionality in the form of inputs (including local state) and outputs.

I'm not sure what's the component under test and what's the dependency here. But I don't see this as particularly complex to test.

Assuming you're writing a client:

- use the client to send a sell order, check that you handle properly the ack (or lack thereof) of the trading platform.

- check that the client handles properly a buy order received from the platform.

- check that the client handles properly a cancel order.

- check that when an order is filled, the client closes it properly after the cancel window is closed.

- check that the client recovers properly after a crash.

To do this, assuming the platform is a dependency of the client, in each test case, you have to implement one or two hardcoded messages from the platform.

all these require state and the syntax & semantics of the exchange, which is not under your control in the real world

> state

You don't need the exchange's state to decide what case you are testing

> syntax

Well, if you don't split parsing in a separate component, yes, you need to test it.

> semantics

You work on your own component's semantics, not the exchange's. You don't need to test that.

> You don't need the exchange's state to decide what case you are testing

yes you do, thd exchange needs to know the state of the trade.

> Well, if you don't split parsing in a separate component, yes, you need to test it

no idea what you mean here

> You work on your own component's semantics, not the exchange's. You don't need to test that.

much the same here. i do not know what the semantics are - it's a black box, with a terribly underspecified API. so i cannot usefuly mock it.

> yes you do, thd exchange needs to know the state of the trade.

You're not testing the exchange, though.

> > You don't need the exchange's state to decide what case you are testing

> yes you do, thd exchange needs to know the state of the trade.

No you really don't.

For your given example, you configure your application state to think it has just submitted a buy order for 500 units. Then, you write a series of tests that check to see what it does if it receives

    - One sell order for 500
    - Two sell orders for 250
    - Five hundred sell orders for 1
    - One sell order for 501
    - One sell order for 250 but nothing else for N simulated minutes
    - Nothing for N simulated minutes
    - The exchange socket closes with an error
    - A sell order for -500
    - A sell order for NaN
    - A buy order for 500
    - A corrupted message
    - etc etc etc

For unit tests you are not checking the full behaviour of the system, you are checking to see how it behaves in a very specific set of circumstances per test.

If you do not know all the messages this API can reply with then just capture a few days or weeks of traffic and you can make a damn good guess from there.

If your code doesn't support this level of isolation then you need to rewrite it until it can, I highly recommend "Working Effectively with Legacy Code" by Michael C. Feathers to help you.

Reimplementing a whole complex API with mocks is not exactly quick or easy or less error prone than the code that should interact with that API.

if your test implements a whole complex API then it's not a unit test

It can be fairly quick with tools like mitmproxy.

The point of mocking is not to reimplement a service you depend on, it is to provide the answer that service would give in the specific example you're testing.

but suppose that service could give all sorts of different responses, depending on (say) the time of day and the country originating the request.

What's relevant to you is your element under test, not the dependency.

Does the component you're testing have a different behaviour for each different response? If yes, you're out of luck (but also, maybe you should split it in multiple components). If not, then you probably don't need to test every possible response the dependency returns.

You're not out of luck. Your testing framework should allow you to hardcode different responses to different requests. Include the time in the response if you need to, it shouldn't be hard.

I've started adding random elements to my mock responses to force other developers to not assert things that aren't relevant to the test.

My problem with these is that they are generally written by the developer who wrote the function.

So devs code the happy path in their function, and test the happy path in their unit test. Yes, in fact, 2+2 is in fact 4. But, do they test add(2,-2) or add(-2,-2) or add(null,2) or add (0,null) .. etc.

Whatever cases they forgot exist and never coded for, never get tested for either. If this is not caught in the initial commit by another dev, then it makes it through to prod happily despite having "unit tests".

This is why I think tests where the application is forced to interact with real world data are better because if your app will be broken by negative/0/null, and that state exists in the real world data, it will break in testing.

Of course this is all a long dissertation on how to code&test the addition of 2 integers. One of the simplest possible functions, now do this on more complex software.

The unit test I think biases later developers to believe the original developer actually coded the software correctly, when to me, in a large number of cases.. it was wrong to begin with, but uncaught.

Reply on news.ycombinator.com