Also, none of the compilers I tried at https://godbolt.org/ were able to combine the 4 s[0...3] 8-bit reads into one 32-bit read, which is somewhat disappointing. They generated 4 separate byte-read instructions, one for each s[0..3]. Yes, I know it could be unaligned, but this is x86 where it would still be faster than 4 separate reads. You would need to do this instead (and sacrifice endianness/alignment-independence):
uint32_t v = *(uint32_t*)s;
*c = (uint32_t)(v & masks[len]) << 18;
*c |= (uint32_t)((v>>8) & 0x3f) << 12;
*c |= (uint32_t)((v>>16) & 0x3f) << 6;
*c |= (uint32_t)((v>>24) & 0x3f) << 0;
*c >>= shiftc[len];
We're still left with a bunch of shifts, ands, and ors, the purpose of which is to essentially "compact" the 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx bitstring into one 21-bit codepoint by removing the bits in the middle. All the compilers I tried weren't so clever as to do this (assuming eax = v from above after masking, and cl = slightly modified shiftc[len]) at any optimisation level: bswap eax
mov ebx, eax
and ebx, 00FF00FFh
lea ebx, [ebx + ebx*2]
add eax, ebx
shr eax, 2
shl ax,4
shr eax, cl
I played around with optimising UTF-8 decoding a while ago, and the above resembles one of the fastest and smallest ways to "compact" the 4 bytes into a codepoint.Of course, this bit string manipulation would be much faster and trivial in hardware (essentially rearranging wires), which makes me want a LODSUTF8 instruction that operates like LODSB but reads a whole codepoint instead, and sets CF on error...