This is pretty easily solved.

"Hey XYZ, it was such a pleasure speaking with you today. I'm going to shoot ya an email in a bit just to follow-up with everything we discussed. Have a great day!"

Or - if that scares you still - just shoot them an email anyway to follow-up and recap without preluding. It gives them the opportunity to change/add/whatever in-text, and they can't say you said something differently because you can point to the email and say - no, we said this (And you didn't correct me).