I always assumed they collect phone numbers this way. People google a phone number and click on a search result. The site can - by looking at the referrer - extract the phone number and infer that it belongs to someone (and use it for any purpose).

It's a similar mechanism that some forums use to highlight the terms that were part of the google query which lead to this site.