Going down the rabbit hole with go-fuzz | Hacker News Reader

Going down the rabbit hole with go-fuzz

mijailovic.net

98 pointsMetalnem14 comments

Compare Go fuzz to a java fuzzer and there are some interesting differences. For example, in the Go fuzzer, it is looking for panics, not errors. An error is okay, it just perhaps isn't interesting. With a Java fuzzer, it is looking for Exceptions, which aren't as clear. If you get an IOException, is that working as intended, or an actual failure? It's much harder to tell. I attribute this not the the fuzzer, but to the language design decisions.

I tried using Go Fuzz on a small parser I wrote, and was surprised that it found a bug. I was quite pleased, but the input was actually valid, my code was just wrong. One shortcoming of Fuzzing is that it doesn't find Type II errors. It can't find invalid inputs that do succeed. For example, if the input to my parser was actually invalid, but the code didn't crash, the fuzzer would happily continue on. I'm glad I fuzzed my code, but no errors found doesn't mean no errors present.

1. Errors can be problems in Go as well if they're swallowed and not handled (for example, writing "os.Chdir("/foo");" will silently cause any errors to be ignored). Note that a Java fuzzer that detects uncaught exceptions would notice a failing chdir() written in this way, while the Go fuzzer would not notice this issue.

2. In Java, it's easy to tell whether, say, an IOException was working as intended or was an actual failure. Just check whether it was caught or not. Granted, that won't handle apps that try to catch wide swathes of exception types, but Go apps can abuse Go's "recover" facility in the same way.

There is a useful linter named `errcheck` for Go which looks for swallowed or shadowed errors [1]. It's included in `gometalinter` [2] (which is worth running as part of the CI checks on any Go codebase).

[1] https://github.com/kisielk/errcheck [2] https://github.com/alecthomas/gometalinter

Unfortunately, there are a number of correct uses of go that swallow errors. For example, fmt.Println returns an error, but you're just not going to see that error handled. Sometimes, it doesn't matter whether something has failed, because there's not a useful, proportionate thing you can do to respond to that failure. And it's not possible to know from machine examination which cases are which. I'm sure for some programs (grep?) a failure to emit a line to stdout is cause for failure. But for most it is not.

`errcheck` ignores unhandle error value of `fmt.Println`.

1. One assumes that os.Chdir is used down the road. If it failed unchecked it will probably cause a panic later on when it is assumed it succeeded. And again, linters help.

2. Apart from the few functions that are only called for side effects, the others must return something. Thus ignoring errors is an explicit action.

I have seen Java apps catch swathes of exceptions more often that Go apps abuse "recover". It is this way in a lot of short examples.

> 1. One assumes that os.Chdir is used down the road. If it failed unchecked it will probably cause a panic later on when it is assumed it succeeded.

You could make the same argument about any errors being ignored. The fact is that syntax/semantics that allows any errors to be silently swallowed is suboptimal.

> 2. Apart from the few functions that are only called for side effects, the others must return something. Thus ignoring errors is an explicit action.

Ignoring errors is not always an explicit action, as you pointed out yourself. In Java, however, it is always an explicit action.

Java is simply more robust at making sure errors are handled than Go is.

Re: recover: yes that's possible but I've never seen it abused like Java's `catch (Exception e) {}`.

If you want this behavior, you can just panic if an error is returned. Or you can divide the set of errors into expected and unexpected kinds, and panic only on the unexpected ones.

But the main reason this is the case is because in Go, errors are things that can arise from correctly behaving functions. If someone passes you invalid input, it's correct to return an error. You wouldn't want this to show up in as bad input in a fuzzer. Most fuzz test cases after all will not be correct input.

Are there any automated tools (especially for Go) for discovering Type II errors?

Unit testing? I think that, from the computer's point of view, type II errors are correct, in that no errors are generated, no crashes, etc. It's a semantic problem, and so human-generated testing (and, of course, end users :-) are the only ways to discover it's wrong.

You would need a method detect them. The question is how to detect them.

One possible solution is to define all valid data by rules. That might be a laborious work. In this case a Fuzzer could check the returned error for valid or invalid input data.

> If you didn’t fuzz it, you can’t say it’s correct.

> — Dmitry Vyukov

I would amend this slightly to state instead "If you didn't fuzz it, you can't say it's [robust]." To invoke correctness seems to suggest that you can do some amount of verification using fuzzing, but that's not really the case (unless you have a reference implementation).

But without fail, fuzzing often evokes a handful of shallow defects. And once those are addressed you can usually dig deeper and find more. I'd be astonished to find a library or program written in C/C++ that has never been fuzzed and undergo fuzzing without any newly discovered bugs.

> To invoke correctness seems to suggest that you can do some amount of verification using fuzzing, but that's not really the case (unless you have a reference implementation).

Not necessarily. You just need to be able to verify correctness of the output and fuzz the following program:

    y := f(x) 
    if (y,x) are not correct then crash

E.g. if you want to verify correctness of the compressor, try to check if a known-good decompressor can always decompress its output to its input. This is simpler than having a reference implementation.

Reply on news.ycombinator.com