This representation is also coming to arrow. But instead of 8 bytes for a pointer it will contain 4 bytes for a buffer index, and 4 bytes for an offset into that buffer. This allows reduced effort when concatenating/merging arrays (no need to copy the data section where the long strings are stored), while remaining a relocatable/serializable format (only offsets, no pointers).
There, the tag that indicates small string mode is not a particular bound on capacity, but the last byte of the struct that is set to zero. That way, the tag also acts as a null terminator for the stored string.
TLDR: you get extra bytes in your small strings
Long version: Copying from FBString code:
struct MediumLarge {
Char* data_;
size_t size_;
size_t capacity_;
};
// sizeof(MediumLarge) == 24
struct FBString {
union {
// For accessing the last byte.
uint8_t bytes_[sizeof(MediumLarge)];
Char small_[sizeof(MediumLarge) / sizeof(Char)];
MediumLarge ml_;
};
};
This has two advantages:
- You get to reuse seven of the eight bytes of `capacity_` in your small string (i.e. your max small string size is 23, not 16 as it would be with a simpler scheme). - You get a null terminator "for free" (though, of course, you still have size and capacity). This is on top of size/capacity.
This could be reduced to 16 bytes by making `size_` and `capacity_` uint32_t instead of size_t.