Hacker News Reader

Comment by dang | Hacker News Reader

Do you know of a way to detect whether an article is the same even if the URL is different? In this case the URLs were:

repost: https://blog.darknedgy.net/technology/2020/05/02/0/index.htm...

original: https://blog.darknedgy.net/technology/2020/05/02/0/

You would have to scrape the site when it's submitted, store more metadata for each submission, do some validation then allow it to post afterwards.

First, the URLs are perfectly identical minus the filename. You could compare paths, minus query string and filename, and that would give you a rough score of likely similarity for further checking. There would be false positives for sites that use the query string as part of their navigation the way HN does.

Then, you could test the social media meta tags (particularly title and author, if it's there), and a hash of the body text (normalized, all lowercase, etc.) for similarity to other possibly similar submissions.

Dupes could be automatically marked as such or flagged for review.

The system you're proposing would likely cause more trouble for the mods through its obscure failure modes than the occasional dupe on the front page.

Yeah, but that's still how you would do it. There's no other or better way to detect duplicates without generating a fingerprint of the content somehow, preferable multiple fingerprints.

You're missing the choice of just not doing it at all, or doing it manually like HN does.

Because dang asked for a way to do better than the way HN does it currently, which precludes that and not doing it at all.

There's the option that there is no way at all of doing it better given the constraints. I'm not saying that's the case, but I'm not seeing evidence to the contrary either.

We spent months working on a system to scrape content when it's submitted. It was utterly impractical. (Side note: I was expecting someone to tell us to "just" check rel=canonical, and had a reply pre-cached: the OP doesn't include that.)

The kind of approach you're describing (compare paths, minus this and that, get a rough score for similarity checking, then a miracle occurs, then you can detect duplicates properly) is a recipe for sinking in the quicksand of corner cases, of which the web has an endless supply. Sorry for being ill-tempered! I don't mean to pick on your response. It's just that we've tried many things in the past. The people who make belittling comments about duplicate detection (not you, the GP) appear to have no idea how difficult it is to get it right, even when you're not dev-resource-constrained as we are.

We also spent months, years ago, working on trying to identify duplicate content on some fingerprinting basis. There was even the fantasy that you could identify two articles that weren't duplicates but were about the same story. This is a research-level problem.

Any real solution here would not be algorithmic, but would rather involve better support for users to supply human curation, e.g. by reporting duplicates that the system missed.

Edit: I see that I've reiterated what majewsky already said, just in a crappier mood.

I thought it might work but I guess I can't argue with "we tried and it didn't," so fair enough.

Reply on news.ycombinator.com