Comment by prohobo | Hacker News Reader

Comment by prohobo | Hacker News Reader

Parent Full thread

Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them.

They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside single batches. So, it could be a long sustained press release.. like propaganda.

As of writing it shows three different stories though:

- 12:15 Will there be a coronavirus relief bill with $1,200 checks?

- 12:30 Nigeria protesters break curfew amid gunfire, chaos in Lagos

- 12:30 Google antitrust case to turn on how search engine grew dominant

Maybe what you're seeing is a bug in the website client, I'll investigate it.

If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;)

Finding duplicate stories that have different texts and titles sounds like something you should use AI for

I'm definitely thinking of ways to improve it for long-term stories. You're right, fuzzy string matching would do wonders for that, thanks :)

I've used trigram matching of article title strings with some success for comparing similarity of news article. If the strings are greater than .6 they are probably about the same topic.

There are news about Pakistan and Hungary in Poland.

Most news articles mention multiple locations. It's probable that those pieces talked about both Pakistan/Hungary and Poland, so will be found in both countries.

The locations are ordered by salience (proximity to start of the article, number of times mentioned), but not completely filtered, so on this specific visualization you will see a lot of matched locations which you wouldn't consider part of the article, but were mentioned in passing.

Trying to understand the content of an article is still really, really difficult, and error prone.

Reply on news.ycombinator.com