If you care about performance, you should consider using Abseil's hash tables instead of unordered_set and unordered_map. The Abseil containers are mostly drop in replacements for the standard unordered containers, except they have a tweaked API (e.g. erase() returning void) that admits important performance optimizations and (except the node based ones) use open addressed hash tables instead of chaining (i.e. each hash bucket is no longer a linked list). You end up with 2x-3x speedups in a lot of scenarios. The standard containers can't be fixed to match the performance of the Abseil ones because 1) the specification API requires pessimization, 2) Linux distributors are reluctant to break C++ "ABI" (such as it is).

I mean, of course you should follow this article's advice on noexcept, but there's a whole world of associative container performance optimizations out there.