Hacker News Reader

Comment by kmfrk | Hacker News Reader

I love RSS, but the one thing that almost makes me anxious is whether I'm going to make one of those mistakes that ends up republishing the whole RSS feed in people's readers.

View on HN

NewsBlur tries to handle this case by comparing stories and if they match up to a certain percentage of the content, considers them duplicates.

Here's the code that does the work:

https://github.com/samuelclay/NewsBlur/blob/master/apps/rss_...

What mistake typically causes this?

I'm still not entirely sure. I think Reddit's had the same issue with some of its feed when subreddits went from private to public, so if even they struggle with it, then it might not be entirely straightforward.

Not even sure RSS readers like Thunderbird have features to deal with duplicates like that.

Just changing the ID of an entry. (Atom: //atom:entry/atom:id. RSS: //item/guid.)

The main time this happens is when people switch website backend or site generator or whatever, and don’t take care to keep the same IDs. In practical terms, IDs are just opaque strings, but they’re supposed to be IRIs (… even though you mustn’t assume it can be dereferenced) and universally unique, so a common approach is to use the page’s URL, which could lead to something like this if you change it:

  <link href="https://example.com/current-title/" type="text/html"/>
  <id>https://example.com/original-title/</id>

Some systems avoid this by using a different form of ID, e.g. WordPress uses its internal post IDs:

  <link href="https://example.com/current-title/" type="text/html"/>
  <id>https://example.com/?p=12345</id>

It doesn’t need to be an https: URL, either; you’ll sometimes come across UUIDs:

  <link href="https://example.com/current-title/" type="text/html"/>
  <id>urn:uuid:01234567-89ab-cdef-0123-456789abcdef</id>

As a concrete example of changing things and keeping old things working, here’s an excerpt from the template for my Atom feeds:

  <link href="{{ page.permalink }}" type="text/html"/>
  {%- if page.year < 2019 %}
      <id>{{ "http://chrismorgan.info/blog/" ~ page.slug ~ ".html" }}</id>
  {%- else %}
      <id>{{ page.permalink }}</id>
  {%- endif %}

An ancient proposal for stable IDs were Tag URIs [RFC 4151]: URIs which still have an understandable identifier like a domain but with a date encoded at the time of minting the URI which keeps the URI "valid" even if the domain lapses and minting "authority" moves to someone different.

tag:chrismorgan.info,2019-01-01:blog/slug

But the real problem, of course, is people caring: You'll need to store the ID with the content and continue using them when moving CMSs or domains. People, apart from your notable exception, don’t do that.

(Sorry for minting an example tag URI in your authority! I shouldn’t have done that according to the RFC.)

[RFC 4151] https://www.rfc-editor.org/rfc/rfc4151.html

Reply on news.ycombinator.com