Hacker News Reader

Look up information cascades.

We're doing something along a similar vein for LazyReadr. The idea is to merge news about the same story together, an important step from there will be deciding which is the authoritative source to display. Going from that to effective search isn't a large leap.

Thanks for the tip. My search turned up this resource, which looks awesome:

http://news.ycombinator.com/item?id=1986198

I suspect Google isn't doing this for legal reasons. They probably have to take care not to look discriminatory in any way.

It shouldn't be difficult to detect. Whichever site receives content first is likely the definitive source.

Also, in many cases (eg. this one) the duplicated content even links to the original source, so even simple pagerank should put stackoverflow significantly higher than its duplicates.

see http://webmasters.stackexchange.com/questions/6556/does-the-... for the primary outcome of this investigation