I'm becoming a fan of the tiny black hole theory. A lot of this seems to point toward a three-way tradeoff between distance, albedo and mass to try to find a sweet spot that explains how something massive enough or close enough could have avoided observation. A small black hole at a pretty significant distance is a rather tidy answer.
That we could be so lucky as to have a primordial black hole close enough to go chuck rocks at.
We're already pretty lucky, if you think about the whole goldilocks thing. Maybe every other species in a habitable system has a little black hole on the edge too.
Since finding it pretty much means being able to see it, would we ever be able to find a tiny black hole? I would expect its size to be sub-kilometer and if it's at its aphelion to be pretty much invisible.
It would be a size of a tennis ball.
So you are saying there is a chance?
(movie reference for humor purposes)
Like another poster said - just chuck a few rocks at it and see which way they curve. That nifty centrifugal launcher would be great for throwing some birdshot in that direction.