Key is not generated on dongle, so you will have it generated and stored initially on PC. By the way any key generation in chip might be VERY insecure, especially with RSA, because chip vendors love to use "backdoored" random generators (CVE-2017-15361).

So, as you use PC(preferable airgapped) for generation and initial dongle configuration, use secure erase after generation and configuration, but if you want, before that you might do backup copy as well to another dongle, or USB flash kept in safe place. There is definitely no plans to make key retrievable from dongle over USB protocol, as trojans might do the same. Still device is not absolutely secure, JTAG access have to be blocked (i have code for that, but didn't tested it properly yet) and for very paranoid people better to cut off jtag electrically, i might publish detailed guide how to do that - later.