Dr. Chaos – A structured fuzzing framework in Nim | Hacker News Reader

Dr. Chaos – A structured fuzzing framework in Nim

github.com

64 pointsplanetis7 comments

Surprisingly, structured fuzzing (mutating inputs to a grammar's generator) has proven less effective than optimized traditional random fuzzing, so far.

https://google.github.io/fuzzbench/ shows that grammar fuzzers (like AFLSmart) find less coverage per minute than standard fuzzers like aflplusplus, even for the cases where the grammar fuzzer has an extremely comprehensive grammar for that format.

Interesting, how does FormatFuzzer or LPM rank? The one is a generator based format fuzzer and the other a mutator based structured fuzzer. And which benchmarks compare any of those two?

I'm not sure! Maybe someone can encourage the authors to submit to Fuzzbench.

My intuition for the results so far is that (a) converting generator functions to bitstrings is very computationally expensive (since the comparison is coverage found per unit of time), and (b) perhaps most memory safety vulnerabilities may be occuring in technically invalid or grammatically inconsistent inputs that the grammar will never find.

I think we need fuzzers for more uses than file formats, like method fuzzing and in that case garbage data just doesn't work.

This is also a testable claim! File formats and RPC do not seem that dissimilar to me, especially when using a fuzzer like aflplusplus, which will automatically build a symbol dictionary by pulling strings out of the binary, and perform some light symbolic transformation (laf-intel and cmplog), see https://lafintel.wordpress.com/2016/08/15/circumventing-fuzz...

It's just sound weird to me because depending of who you ask, they will tell you generator fuzzers work the best. However defining what's a good benchmark and what to measure other than code coverage is even more complicated. But I guess we are talking about google they might have figured it out.

Yes, that's why I described the empirical results as surprising.

Reply on news.ycombinator.com