...if after swapping buffers around, our input cursor hits the mark again, we just hit the refill path again to generate more zeroes. (This is why the copying code uses memmove).
This misses one more optimization: if you consume bits by shifting them off the end of your word, then you are shifting in zeros in their place. If your semantics are that all reads past the end of the buffer return zeros, then once you have hit that end, you can just set bitcount to LOTS_OF_BITS (an arbitrarily large constant).
That is, you pretend your 32-bit word has a lot more than 32 bits in it. Then you're extremely unlikely to ever hit the refill code again. For extra points, you can make the constant something that can easily be represented as an ARM immediate, like 0x40000000.
We do this in the bitreaders for Theora, Opus, and Daala (thanks to Simon Hosie for contributing the idea to Theora back in the day).