Hacker News Reader

Comment by parentheses | Hacker News Reader

Very specific question: when tests fail due to a single example, you suggest creating a specific assertion with the given seed.

I imagine the seed is used to generate data and depending on the order of your generators, it produces different results.

For example, in:

    PBT.assert(seed = ..) do
      PBT.property(PBT.integer, PBT.string) ...
    end

Would changing the order of parameters to `property` change the actual test case?

View on HN

(not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.

So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape.

@OP:

I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing.

The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case.

Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.

I think there are usually three actions a programmer can take when PBT fails.

- Create a test case that doesn't depend on PBT as you suggest.

- Fix the production code being tested since its failure is an unexpected bug.

- Fix the PBT itself. This means that the programmer has had wrong assumption for the test target.

I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)

>yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.

Correct. The test inputs are determined by a seed and generators (including the order of generators).

Reply on news.ycombinator.com