Hacker News Reader

Comment by ajmurmann | Hacker News Reader

I had a conversation with a coworker a few days ago where I jokingly suggested building an AI system that you give a failing test suite and it uses stackoverflow answers to make your tests pass. Sounds like we are one step closer to making that happen.

View on HN

Have you heart of StackSort?

https://gkoberger.github.io/stacksort/

Original credit goes to the hovertext of https://xkcd.com/1185/, I believe.

Yup, I stole the idea from there :)

That actually sounds realistic. A well defined test suite is probably a good target for AI. Main issue is no partial wins, which are needed for training, so you'd need to break all functionality down into absolutely minimal units.

Also if you overfit, your code is going to look like:

  int add(int a, int b) {
    if (a == 1 && b == 1) return 2;
    if (a == -1 && b == 0) return -1;
    if (a == 13 && b == 7) return 20;
    return 0;
  }

But at least it passes my unit test suite!

Just rerun it with a lower target code complexity. You'll know you set it too low when it starts monkey-patching the test framework to avoid actually doing anything.

That's why you use property based testing like Python's Hypothesis or QuickCheck; instead of example based unit testing.

This can be avoided by optimizing for the simplest code that works. Checking many cases in multiple lines is always going to be longer and more complex than just a single operation on a single line.

Which is how AIs avoid overfitting, in general, by penalizing complexity.

Reply on news.ycombinator.com