Hacker News Reader

Comment by mmmkkaaayy | Hacker News Reader

There wasn't much info on the GitHub repo about how it actually works. Here's the paper from the authors: https://arxiv.org/abs/2007.14049

From what I gather from the paper, they frame the problem of test generation as a search problem. An evolutionary algorithm randomly mutates a randomly generated test suite. The evolutionary algorithm optimizes for greater branch coverage.

Excerpt from the abstract:

"Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and can even alleviate the problem of absent type information to some degree. However, our results demonstrate that dynamic typing nevertheless poses a fundamental issue for test generation, suggesting future work on integrating type inference"

View on HN

I have trouble even understanding the point of this. To me "automatic test generation" seems as bad an idea as "automatic code generation".

The point of tests for me is to express intent for what the software should do. The code then expresses the details of how we do it.

I could see a use for evolutionary algorithms to probe code for hidden bugs. But even there it seems limited. I definitely see a use for things like Hypothesis, which makes test expression more powerful: https://hypothesis.readthedocs.io/en/latest/

It makes sense to me that this comes from an academic perspective, and not from people who actually make software for a living.

Automatic test generation in this case is not testing your hypothesis of how the code behaves, but rather records how the code actually does.

This can be very useful if you are dealing with legacy code bases or just don’t want to write tests yourself.

I agree that if your are going to write the tests, using hypothesis is a very wise decision.

Even there, the code already records how the code behaves. The question is always which of those behaviors are intentional, meaningful. I am unable to fathom the utility of recording an arbitrary slice of behaviors, because those are very likely to be orthogonal or even contrary to the actual intent of the system. For example, every bug is part of how the code actually behaves.

I would much rather inherit a code base with zero tests than ones automatically generated by people who "just don't want to write tests".

A test fixes the behavior. When later the need to change the code arises, you have a meaningful way of comparing results.

Any test is better than none, I’m my opinion.

That's definitely not true. Tests that lock in a bug or an accidental epiphenomenon of code are demonstrably worse, because they actively discourage improvement.

Reply on news.ycombinator.com