It could also simply have been impotence that he did not want to acknowledge.

Made perhaps even more likely with Antoinette's alleged proclivities, her children's questionable paternity, and so on.