I've had this problem quite a lot with both the default keyboard and with GBoard, although seemingly a little less frequently with GBoard. I also find that the speech-to-text often fails to grasp the contextual clues and blatantly uses the wrong "your" or the wrong "their."
I hadn't previously thought about it, but it appears that Gboard is the default on my phone.