Securely deleting the keys is safer than simply practically inaccessible. That suggests their is some impractical way to access the data, care to suggest what that might be?
Personally, that's not a risk that I would care about (especially in the context of an Echo Dot or even an iPhone), but deleting the keys will not be better than inaccessible forever.