So, what is the right way to deal with gamma in a graphics application?
Gamma encoding is only really necessary to optimize around the limitations of 8-bit color channels. So, load sRGB888 convert to linear 32-bit floats, do linear algebra, convert back to sRGB888, store the results.
And 16 bit floats are plenty, so that's a good option as long as the hardware won't do something dumb with them that makes them slower.
I guess ideally you'd reallocate the sign bit to the mantissa, and maybe an exponent bit too.
A gamma of 2.2 puts that '15' at 0.2% brightness, and the '20' at 0.4% brightness.
An 8 bit linear representation would make that '20' square the minimum brightness above zero. The next step up would be roughly the '30' square.
So yes, the gamma curve is very necessary. Even 12 bit linear would be a bad idea. So 14 or 16 minimum. And adding HDR is something like 2 bits with gamma and 8 bits without.