Hacker News Reader

Hacker News

Comment by beders | Hacker News Reader

Full thread

beders

Ah, hacker news, the web site that can't detect duplicates or afford a message preview.

Anybody else amused that we use this bare-bones, no-frills forum to discuss high-tech? ;)

View on HN

Do you know of a way to detect whether an article is the same even if the URL is different? In this case the URLs were:

repost: https://blog.darknedgy.net/technology/2020/05/02/0/index.htm...

original: https://blog.darknedgy.net/technology/2020/05/02/0/

You would have to scrape the site when it's submitted, store more metadata for each submission, do some validation then allow it to post afterwards.

First, the URLs are perfectly identical minus the filename. You could compare paths, minus query string and filename, and that would give you a rough score of likely similarity for further checking. There would be false positives for sites that use the query string as part of their navigation the way HN does.

Then, you could test the social media meta tags (particularly title and author, if it's there), and a hash of the body text (normalized, all lowercase, etc.) for similarity to other possibly similar submissions.

Dupes could be automatically marked as such or flagged for review.

The system you're proposing would likely cause more trouble for the mods through its obscure failure modes than the occasional dupe on the front page.

Yeah, but that's still how you would do it. There's no other or better way to detect duplicates without generating a fingerprint of the content somehow, preferable multiple fingerprints.

You're missing the choice of just not doing it at all, or doing it manually like HN does.

Because dang asked for a way to do better than the way HN does it currently, which precludes that and not doing it at all.

There's the option that there is no way at all of doing it better given the constraints. I'm not saying that's the case, but I'm not seeing evidence to the contrary either.

We spent months working on a system to scrape content when it's submitted. It was utterly impractical. (Side note: I was expecting someone to tell us to "just" check rel=canonical, and had a reply pre-cached: the OP doesn't include that.)

The kind of approach you're describing (compare paths, minus this and that, get a rough score for similarity checking, then a miracle occurs, then you can detect duplicates properly) is a recipe for sinking in the quicksand of corner cases, of which the web has an endless supply. Sorry for being ill-tempered! I don't mean to pick on your response. It's just that we've tried many things in the past. The people who make belittling comments about duplicate detection (not you, the GP) appear to have no idea how difficult it is to get it right, even when you're not dev-resource-constrained as we are.

We also spent months, years ago, working on trying to identify duplicate content on some fingerprinting basis. There was even the fantasy that you could identify two articles that weren't duplicates but were about the same story. This is a research-level problem.

Any real solution here would not be algorithmic, but would rather involve better support for users to supply human curation, e.g. by reporting duplicates that the system missed.

Edit: I see that I've reiterated what majewsky already said, just in a crappier mood.

I thought it might work but I guess I can't argue with "we tried and it didn't," so fair enough.

IMO, HackerNews has a few frills, mechanics-wise. Like the fact that you can't downvote direct responses to your comments. Or how you can't downvote at all until you get the right amount of karma. I also consider the lack of notifications about new responses a feature, but a lot of people will probably disagree.

@ainar-g: “IMO, HackerNews has a few frills, mechanics-wise. Like the fact that you can't downvote direct responses to your comments. ”

I would prefer a voting-up only option, else the down-voting option gets abused.

Well that's why down-voting requires a karma threshold before one can actually downvote

Sadly it is easy to gain karma once your comments amplify an existing echo-chamber e.g pro mozilla/rust comments

@beders: “Ah, hacker news, the web site that can't detect duplicates or afford a message preview.”

They could provide a report a duplicate link.

@beders: “Anybody else amused that we use this bare-bones, no-frills forum to discuss high-tech? ;)”

I find it one of, if not the best tech forum online. They could stop modding comments that they disagree with into oblivion.

It's still the best place around.

A lesson that "more tech" ISN'T always the answer.

Isn't always the answer doesn't mean isn't ever the answer. This is the site where you are told you can't post another message only after you write and submit it.

Sounds like it slows things down quite a bit.

That's intentional.

For me it's honestly refreshing and I wish more sites were like this.

Simplicity is the ultimate sophistication. -someone else i think

From th HN FAQ:

> If not, a small number of reposts is ok.

This one violates the 'if not' though.

Why implement a new duplicate detection feature when their well-trained neural net seems to be working just fine??

https://xkcd.com/2173/

Duplicates are allowed and often stories are reposted on the weekend.

Why would people who read source code for a living need a frilly marketing puff type site? Slashdot is still around if you wanted something different.

Duplicates are only allowed if a story has not had significant attention in the last year or so: https://news.ycombinator.com/newsfaq.html

This one did though: https://news.ycombinator.com/item?id=23062072. In such cases we add [dupe] at the top and bury the thread. That's not because we think it's a bad article or a bad thread! It's because front page space is the scarcest resource HN has (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...) and curiosity withers under repetition (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...).

Reply on news.ycombinator.com