On the profile page of your contact there is a "★ Relationship" button. Click that and it will let you add general notes or more detailed information about how you met.
I don't want to put in "obnoxious guy from conference" and then have that person be able to read it.
"Your note will be visible in the Relationship section of the profile and will only be visible to you."
Of course you may wish to store such information in a different system to avoid the possibility of LinkedIn inadvertently leaking the data.
[1] https://help.linkedin.com/app/answers/detail/a_id/43370/kw/W...
But, then that would be work.