Hacker News Reader

Comment by mjs | Hacker News Reader

Interesting position on testing: "I generally don't consider my own software adequately enough tested until its tests have turned up a bug in a compiler/toolchain. So far I've not encountered a compiler bug for libsecp256k1 [...] so I may have to settle for discovering that a ubiquitous system library couldn't square correctly."

(The bug was found via comparing the output of libsecp256k1 and OpenSSL on "specially-constructed random inputs.")

View on HN

For those not familiar with Greg's work, this presentation on the testing he did for the Opus codec is informative: https://www.ietf.org/proceedings/82/slides/codec-4.pdf

Particularly slide 20.

Comparing multiple implementations is a great way to find bugs. Even running the same code with different compilers or platforms helps find bugs.

It doesn't find cases where they are both wrong, but often finds cases where one of them is wrong.

It is a very common testing strategy with maths libraries.

You might be intrested in this [1]. Its a 31C3 talk about exactly this. There idea is to write a runable spec (that only focues on correctness, instead of speed and things like that) that then can be used to test against real world application.

There idea is to do this against all the MAJOR interfaces that we all depend on TCP/IP, x86 memory model and stuff like that.

[1] http://events.ccc.de/congress/2014/Fahrplan/events/6574.html (the talk is on youtube or on ccc video streaming side)

Runnable/verifiable specs are a good idea. A colleague of mine did some great work with compiling the H265 & VP9 specifications to produce a test suite, in a similar manner to fuzzing:

http://www.argondesign.com/products/2014/may/27/streamsvp9/

It would help if the videos were automatically linked from the event page:

http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_...

The talk on youtube: https://www.youtube.com/watch?v=ca0DWaV9uNc (I couldn't find a link on that page)

That 'CCCen' account is an imposter and evil. Please do not link to it or the companion 'CCCdeVideos'. Some background on this issue: https://events.ccc.de/2015/01/03/the-youtube-and-stream-dump...

Here is the official CCC channel: https://www.youtube.com/user/mediacccde

The official recording: http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_...

The official recording on their official YT channel: https://www.youtube.com/watch?v=MBIHPLFmcgA

> instead of speed

Not when developing crypto. You have to look at execution speed, too, to prevent timing attacks.

Execution speed variance

It doesn't matter if it takes 1s or 10s, but that it takes 10s every time (to verify a password, for example)

Why would that matter in the case of testing for correctness? I thought these 'runable spec' implementations were only to verify that the output of the 'real' implementation is correct and thus would never even be used anywhere that could/would be attacked. Am I not understanding the purpose correctly?

When testing for correctness why would you care about timing attacks? You would obviously have multiple tests covering correctness, performance, security, etc.

That's only because author uses a pretty standard compiler. Try building your flawless library on ARM, Newlib, etc. and you're pretty much fighting your tools all the time. The first compiler/toolchain bug is a rite of passage, yes, but not a measure of the code quality (although they do correlate).

It does say "a compiler/toolchain". Opus has both fixed point and floating point implementations. The fixed point one was extensively tested on ARM too, I'm sure.

Reply on news.ycombinator.com