I believe that this (SWAPGS) vector is impractical but previous spectre proof of concepts exists if you look hard enough.
I don't accept the general hand waving "they keep getting better" because they haven't.
Most of these the exfiltration rate is so slow that simply xor'ing the secret with a random seed that changes locations or moving the memory around itself would prevent any attack.