I stared at the code for some time and couldn't see any mishandled edge cases. It was then I realized that I could automatically generate an arbitrary amount of unit tests, because it's simple enough to call strcat and encode that to get the correct result. So, that's what I did: generate two strings of ACTG, encode them, concat, and compare against the result against that of the encoding of the two strings concatenated. Sure enough, a mishandled edge case popped up after trying a couple hundred strings. There was probably no way I could have found that bug without this approach.
It's a really simple concept, but you need a couple of things: first, a very wide input space. Next, you need an alternate way of verifying the result. If you just want your program to not crash, you get this for free. In my case above, I could apply the two functions (concatenate/encode) in the other order to easily calculate the expected result. If you have these two things, a fuzzer is the logical thing to write to make sure your code is rock solid.