Optimized Go gzip/zip packages, 30-50% faster
blog.klauspost.com
blog.klauspost.com
Deleted comment
I went to the test subfolder:
https://github.com/klauspost/compress/tree/master/testdata
I saw 3 files.
The kind of test data I'd like to see for something like a compression library would be:
0 byte files
1 byte files
2,147,483,647 byte file
2,147,483,648 byte file
2,147,483,649 byte file
(to check for bugs around signed 32-bit integers)
4,294,967,295 byte file
4,294,967,296 ...
4,294,967,297 ...
(to check for bugs around unsigned 32-bit integers and see if 64bit was used when necessary for correctness)
Also add files of sizes that straddle the boundary of algorithm's internal block buffers (e.g. 64kb or whatever). Add permutations of the above files filled with all zeros 0x00 and all ones 0xFF. I'm sure I'm forgetting a bunch of other edge cases.
The programmer may have done a wonderful job and all the code may be 100% correct. Unfortunately, I can't trust a library replacement unless I also trust the test bed of data that was used to check for defects. It's very common for performance optimizations to introduce new bugs so there has to be an extensive suite of regression tests to help detect them. Test data for bug detection has a different purpose than benchmark data showing speed improvements.
Those multi-gigabyte files are not git repository friendly so perhaps a compromise would be a small utility program to generate the test files as necessary.
However I believe even better testing would be to fuzz it for a few hours [3].
[1] http://golang.org/src/compress/testdata/
[2] https://github.com/klauspost/compress/tree/master/zip/testda...
Deleted comment
I don't think that streaming vs file is relevant. The deflate algorithm (stream) may have no global knowledge of the eventual file size but the gzip file format does.
Anyways, even with a cursory google search, we can find a compression bug related to file sizes:
https://bugs.openjdk.java.net/browse/JDK-6419239
As for an example of streaming bugs, zlib's creator Mark Adler reported that Microsoft coded its deflate implementation incorrectly.
http://stackoverflow.com/a/11435898
The lesson is that re-implementing even well-known algorithms such as zlib/gzip/pkzip is not trivial.
Yes, you are not criticizing the programmer's effort. That's what a pull request is for. A pull request with a failing test in it would be some very concise criticism.
thanks