Comment by Vinnl | Hacker News Reader

Comment by Vinnl | Hacker News Reader

I've been aware of property-based testing for a number of years now, but never had a good opportunity to give it a try. Then the past year I had a piece of serialisation/de-serialisation code, which was the perfect opportunity for a rather simple property-based test. That gave me the hang of it, and found two (minor, but still) bugs.

Then recently I had a fairly larger, more error prone piece of work that lend itself very well to property-based testing, and it's been a godsend. It helped me discover a number of bugs, this time with the risk of causing privilege escalation. And since the proptests started succeeding reliably, I've been very confident that a rather complex piece of code now actually does what it's supposed to.

If you're working in JavaScript, I can recommend fast-check [1].

Another interesting approach, that I haven't yet tried, is Quickstrom [2], basically Puppeteer for proptests. It opens a webpage in a browser, performs some random interactions (pressing buttons, entering data, etc.), and then verifies that properties you specified still hold.

[1] https://dubzzz.github.io/fast-check.github.com/

[2] https://quickstrom.io/

I was hoping from some insight, but as happens every time I decide to try property testing, the examples on your links are trivial to either enforce the property at type or construction, or to separate it from the rest of the code so that their implementation becomes at least as obvious and error-prone as the tests.

Property based testing looks like a really good idea, but I never gained anything from applying it on practice. There are probably some application domains that they are good for, but I still didn't find them.

It's great for compiler testing.

The first approach is to generate random correct programs and see if they do the same thing with different compilers and/or different optimization flags.

The second approach is to take a correct program, note that some parts of it are not executed on some particular input, then mutate that part and run on the same input. The output should be the same.

Also, any program (not just compilers) should satisfy the properties that no assertions fail and that no sanitizer failures occur.

Property-based testing is good when there is a property to assert that emerges in a non-trivial manner from the code under test. If you do not have such a problem, property-based testing provides no value over a simple, example-based test.

In the case of the parsing code from the article, the emerging property is that a serialization followed by a deserialization should always yield the original result.

In the case of the 'binary or not' case from the article, the non-trivial, emergent property is that the function never fails with an exception.

Most modern software development is plumbing, stitching together platforms, libraries and frameworks, and there are rarely non-trivial, emergent properties where property-based testing is useful.

Obvious code should have obvious properties, in which case you should be able to gain value from property tests. If you make every little function high-quality and well tested, you should end up with better code overall. The only functions, IMO, that don’t deserve properties as much as functions which are exclusively the composition of other functions without any other logic. In that case, the function is often, by definition, its result.

here's an example I had a few month ago: Password Requirements, merging of requirements and generating passwords that fulfill the requirements:

- merging/combining of requirements is a monoid operation with an "empty requirement" (e.g. one that accepts every password) as the neutral element. Finding this requirement, I could write properties for the monoid properties (a + b = b+ a, a + 0 = a, and so on) - when generating a password from a requirement, the same requirement must accept the generated password (`requirement.accept(requirement.generate()`)

In general: if you find mathematical rules to your code (commutativity, associativity,. ..), these make a great starting point for property tests.

If you're open to Erlang/Elixir I liked [0]. Obviously a bigger commitment than some blog posts and tutorials, but worth it in my opinion. It more clearly (to me) presents the case for property-based testing and goes through more complex examples than most blog posts which help to illustrate the utility more effectively.

[0] https://pragprog.com/titles/fhproper/property-based-testing-...

I had a similar experience after watching a Computerphile video with John Hughes, one of the authors of the original propert-based testing tool, QuickCheck [1]. Running hundreds of thousands of unique tests and finding the simplified cases was mind blowing to watch.

I loved the video but my first experience with this method was with Python's Hypotheis [2], I haven't used it a ton but it's great for finding parsing errors. In the words of Python core developer Raymond Hettinger:

> It's not quite fuzzing, but it hits it with the kind of test cases that a good QA engineer would typically come up with, and it does it in automated fashion.

1. https://www.youtube.com/watch?v=AfaNEebCDos

2. https://hypothesis.readthedocs.io/en/latest/

3. https://youtu.be/ARKbfWk4Xyw?t=319

I checked out Quickstrom, and I thought "Wow, this looks amazing!"

Then I noticed who wrote it and thought "Ah. Well that makes sense."

Reply on news.ycombinator.com