They did make everything slower than it used to be when it comes to new UI elements, but the context menu can also be slowed down by third parties.
On my machine, the primary cause was the shortcut to AMD radeon settings. You can remove it with this powershell command :
Get-AppxPackage -AllUsers AdvancedMicroDevicesInc-RSXCM | Remove-AppxPackage -AllUsers
It was a major pain to find how to disable this thing. In fact, if you google "AdvancedMicroDevicesInc-RSXCM" (with the quotes, to get a verbatim search) you won't find more than a handful of results talking about its existence.
It doesn't uninstall the settings panel itself, just the shortcut. Yes, the context menu shortcut is a standalone app package of its own. It's crazy.
After uninstalling this AMD crap the context menu went from taking a second to being almost instantaneous.
On other people's machines I've also remarked certain antivirus like Kaspersky causing major issues with 11's context menu performance.
I thought the new context menu was all about fixing the mistakes of the previous ones when it came to the nasty things people can do in extending explorer.exe but seeing the behavior of those new third party additions to the menu, I guess not.