I don't know how actual asymmetric KDFs work, but I was thinking more along the lines of secret -> private key -> public key, i.e. the public key doesn't contain information about the secret.