Back in the day I'd suggest doing it via Alexa top sites, but now that Alexa is gone, I'm not sure what strategy I would use, but I would want to hit sites that are like the "top 10000 most popular" first, and scrape every inch I could.
They try to highlight small, personal websites instead of the big mainstream sites.
(This was a HN submission 2 weeks ago)