No, those registers are per-thread. The only things that are per-process are the page tables and, in special cases, CR4 and LDTR. If you want to see the code on Linux, look at switch_mm_irqs_off().
> Context switching when dealing with threads is typically done in software, not in hardware
Hardware context switches (the x86 "task" mechanism) have been obsolete for a long time and are unavailable in long mode.
> precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored.
And that's the problem: the OS doesn't know that the registers are unused until it saves them.
> Saving/restoring registers isn't even the expensive part of a context switch anyway. Switching address spaces, cache flushes/misses, etc... are all more expensive than saving/restoring registers.
Go compare WRMSR to MSR_FS_BASE to the cost of a CR3 (page table) write. You might be surprised. And that CR3 write is about to get considerably faster, maybe in Linux 4.13, when my PCID patch set lands. (Yes, Linux will eventually use WRFSBASE instead, but that's a minefield and will take a while.)