It's more likely than you think.

"Oral herpes infection is mostly asymptomatic, and the majority of people with HSV-1 infection are unaware they are infected." [1]

[1] From https://www.who.int/news-room/fact-sheets/detail/herpes-simp...

I'll announce my scarlet letter to make a point. I was recently tested for HSV and came back positive for HSV-2. I've never been tested before so I have no idea how long I've had it, but it was a surprise because I've had zero symptoms. In fact, I figured it was a false positive so I got re-tested.

>I suppose it's remotely possible that I contracted HSV since then without ever having any symptoms

It actually is. Most people are asymptomatic, or their symptoms are so mild that they confuse them for a pimple.

https://www.cdc.gov/std/herpes/stdfact-herpes-detailed.htm