This is a good point but your argumentation isn't really convincing. Here's a simple fix for your scenario: only use talks in your circuit as tie breaker. The other concern("possibly even the less qualified person") is an impossibility given your assumptions("all else being equal between speakers"). If you have a perfect meritocracy, and only use your own tests to measure merit, then you would maintain it.
Of course, as you said, achieving perfect meritocracy is impossible, which means we should always consider how to work around the flaws in our existing approach. I like the idea of randomization, but you have to consider that it has its own flaws - drawing from a small pool wouldnot have the desired diversifying effect, but drawing from a too large pool might introduce too many students that are not at the same academic level, with all the problems that brings. In a sense, random choice has similar problems of not being able to achieve "perfection" as meritocracy, and they would need to be balanced together.
To me, all of this concern over the admissions process of the top tier schools tells me that there's a growing need for an even more elite tier of schools. Those schools will undoubtedly have a diversity problem, but it does not sit right with me to limit the ceiling for the most academically gifted students just because most of them come from a privileged background. What this tends to do is exacerbate the problem, as the more wealthy can afford more expensive, privately tailored programs to their kids - the brilliant underprivileged kids will not have this opportunity. Just like we'll never achieve perfect meritocracy, we'll never eliminate privilege - our pursuit of equality should be tempered by the same concerns of unintended, negative consequences due to our imperfect approach.