That’s not quite true, though. What is true is that the server does not have a plaintext copy of your private keys. That’s a crucial difference.
The server has an encrypted copy of your private key, which your with token decrypts with its private key. That is how a usb key can store an unlimited number of U2F credentials: The USB key isn’t storing them at all. They are stored on the server you are authenticating to.
What this also means is that if there is a backdoor in the crypto algorithm used by your token and an attacker gets a copy of the server’s U2F credentials, then you are at risk of having your private key stolen.