Bugs caused by misunderstood or unexpected interactions between subsystems and specification bugs seem to predominate these days IME but discussion about sophisticated ways of dealing with these things seems fairly scant.
Bugs caused by misunderstood or unexpected interactions between subsystems and specification bugs seem to predominate these days IME but discussion about sophisticated ways of dealing with these things seems fairly scant.
In my experience working on many very large software, I have encountered one that was head and shoulder above the others in quality. This is not just a hunch. There were metrics tracked, thing like MTBF, and percent of sessions with clean exits, etc and this software was ridiculously better than the others.
The difference lied entirely on the fact that the better software had almost no unit test, but had 21K+ integration tests. I've worked on other software that also had more integration tests than average, and they too were of better quality.
The problem is making integration testing easy needs to be designed in from the start and requires not just one simple trick but multiple inter-locked design decisions that accumulate into making integration testing easy. In this particular case, one could write a new test just by using the software and a new test would be generated automatically. That's how you achieve writing 21k integration tests.
There are many unit test framework, but I've yet to see a generic integration framework. The reason is that integration tests are run with the whole software, so the whole software needs to be designed around them.
I'm interested in the topic. Could you talk more about it, or point to some resources describing in more details how this was achieved/what decisions had to be taken?
They might be a maintenance nightmare but... they'll probably be better than no tests
I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are
00000000,00000000,00000000,0PPPPPPP
00000000,000000QQ,QQQQQQQQ,QQQQQQQQ
The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ
Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make.The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling)
int pw = bitWidth(p);
int qw = bitWidth(q);
int pwr = (pw%8);
int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0);
byte[] data = new byte[totalSize + 1];
int b = data.length - 1;
while (p > 0) {
data[b]=(byte)(p&0xFF);
p = p >>> 8;
if (p != 0) {
b--;
}
}
if (pwr != 0) {
while (q > 0) {
data[b--] |= ((byte) ((q << (pwr)) & 0xFF));
q = q >>> (8-pwr);
data[b] |= (byte) (q & 0xFF);
q = q >>> pwr;
}
}
else {
while (q > 0) {
data[--b] |= (byte)(q&0xFF);
q = q >>> 8;
}
}
data[0] = (byte) pw;
return data;
It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question).
Most of the data is kept in contiguous memory mapped byte arrays.
This is hot code with an invocation count in the billions, so among the requirements is that it shouldn't allocate objects while at the same time being thread safe. That's a tricky combo.
Every while could be it's own function and now you have a bunch of simple unit tests. Unit tests are literally the double accounting for the logic, which is the value.
Any unit tests for that code is going to be horrendous to get decent coverage...and this is how it's easy to know it's too complex. Functions, especially if they are small, can be considered free, given the JVM inlines them into branches anyway....but I understand this is a hot path. If you want to avoid re-allocation, Java gets into your way. You could split up the function and use a singletonClass for holding the array, without having to pass the array by value and still being able to reference the singletonClass...blah blah blah.
> Uncle Bobbing this type of code doesn't make it much clearer
I get it and agree, while still recognizing the problem. Not you not the code not me. Java.
It's really unfortunate that Java has damaged programming practices so subtly and completely by being unnecessarily byzantine AND pushing horrible solutions like Spring.
bit data[length];
int width = 6;
bit value[width] = random(64);
for (i=0; i < lim; i += width) {
int j = endof(i, width);
data[i:j] = value;
}
or something like uint8_t c = 0xff;
c[0..3] = 1;
c[5..7] = 1;
// c == 0x31Otherwise, `std::vector<bool>` _may_ be specialized to occupy one bit per element [1].
[0]: https://en.cppreference.com/w/cpp/utility/bitset [1]: https://en.cppreference.com/w/cpp/container/vector_bool
public static byte[] func( uint p, uint q )
{
int pw = 32 - BitOperations.LeadingZeroCount( p );
int qw = 32 - BitOperations.LeadingZeroCount( q );
Span<byte> buff = stackalloc byte[ 9 ];
ulong val = (ulong)q | (ulong)p << qw;
BinaryPrimitives.WriteUInt64BigEndian( buff.Slice( 1 ), val );
int totalBytes = ( pw + qw + 7 ) / 8;
buff[ 8 - totalBytes ] = (byte)qw;
return buff.Slice( 8 - totalBytes ).ToArray();
}
In C++ about the same, albeit you gonna need either C++/20 language version, or some non-standard compiler specific intrinsics for bit scan and byte swap.To exploit this i had to use cython from python to define a c function which used a gcc compiler intrinsic to allow me to write the asm instruction inline. It turned out about 7 lines of code so it wasn’t as bad as it sounds.
In your case, you’d need to use JNI i suppose so it might not be any less code but it should at least be a lot faster if that matters.
ZZZZZZZZ,0000000Q,QQQQQQQQ,QQQQQQQQ,QPPPPPPP
With Z still being the width of P?Here's my attempt, which is shorter, and i think quite a bit clearer:
int pw = bitWidth(p);
int qw = bitWidth(q);
int qpw = qw + pw;
long qp = Integer.toUnsignedLong(q) << pw | Integer.toUnsignedLong(p);
int qpwBytes = ((qpw - 1) / Byte.SIZE) + 1;
byte[] bytes = new byte[qpwBytes + 1];
bytes[0] = (byte) pw;
for (int i = 1; i < bytes.length; i++) {
bytes[i] = (byte) (qp >> (qpwBytes - i) * Byte.SIZE & 0xff);
}
return bytes;
This did take me quite a bit of fiddling to get right - but it wasn't the bit-twiddling, it was the index arithmetic for copying the bytes into the array!https://docs.oracle.com/javase/7/docs/api/java/math/BigInteg...
Testing is not a panacea, by design you can only test for what you know and predict. In addition to testing, your setup needs to be resilient enough that unpredictable bugs can be solved quickly and won't lead to data loss.
I've also noticed a parallel phenomenon where teams double down on unit tests for branch conditions, coverage %, etc. when the bugs that are really hurting them predominantly cant even really be detected by unit tests.
My goal build pipeline brings up the app in docker-compose, then randomly kills services while running integration tests. So many failures are because people don't consider the happy path in a distributed/highly modular system.
Fuzz testing is still a type of testing.
Could fuzzing activate other kinds of bugs? Sure. But it's not magic. It's mostly working on known kinds of bugs.
If you put asserts everywhere in your code, to test all assumptions the code has, then fuzzing is an excellent way to see whenever one of those assumptions is violated because those violations are turned into crashes.
In my experience, fuzzing with extensive asserts is one of the best way to suss out bugs. It gets even better if you can take all of the paths the fuzzer found and run them through something like ASan or Valgrind.
You can read more about that at [1].
[1]: https://git.yzena.com/gavin/bc/src/branch/master/manuals/dev...
Programming errors are of special interest to programmers, as we tend to judge each other a bit harshly when these errors emerge as the problem is not on the business side or other teams making requirements.