Accessing a giant lookup table is about the most cache-unfriendly thing you could do. This is not a good approach when latency matters.

Smaller LUTs are fun to think of, especially if you eliminate the lossless angle. I'm really not that interesting to watch during a videoconference. When I'm not talking, it would probably be better for viewers to make up a stream of me sitting still with a dumb look on my face. It should be loosely matched to my actual position/pose, so that when I start talking or my dog attacks me or the bomb explodes and wipes out my room, the transition back to actual data is relatively seamless.

(And if you set the threshold high enough, maybe I don't need to be in the room at all...)