jungsteven

Appreciate the feedback and suggestions!

Thanks for sharing the papers! I remember reading the first one from Google and can’t wait to dive into the new one. Appreciate the insights!

+ it would be great to see more research comparing LLMs and traditional methods in terms of cost-effectiveness. In my opinion, regarding the cost issue, there are engineering ways to mitigate this, such as running the LLMs only on changed code.

Nice question! The paper acknowledges that LLMs generate mutations slower and are more costly than traditional methods like PIT and Major, which are traditional testing tools. They did include metrics like cost per 1K mutations. However, the researchers focused on the effectiveness and high quality of the mutations generated by LLMs. For instance, GPT-3.5 achieves a 96.7% real bug detectability rate compared to Major’s 91.6% (not to mention GPT-4 outperformed all of them). All in all, LLMs produced fewer equivalent mutants, mutants with higher fault detection potential, as well as higher coupling and semantic similarity with real faults.

Hey engineer here. Mutation testing may not be a familiar concept. To put it simply, it tells you how effective your unit test cases are at catching faults by injecting faults into your codebase.

If you are interested in learning more about mutation testing and how big tech companies use it, read:

- State of Mutation Testing at Google: https://research.google/pubs/state-of-mutation-testing-at-go... - Industrial Application of Mutation Testing: https://homes.cs.washington.edu/~rjust/publ/industrial_mutat... - LLM-based Mutation Testing: https://arxiv.org/pdf/2406.09843 - Medium Blog on Mutahunter: https://medium.com/codeintegrity-engineering/transforming-qa... - Short Demo: https://www.youtube.com/watch?v=8h4zpeK6LOA

Feel free to ask me any questions. My wish is to get mutation testing widely spread!