Full threadcrystalgiver·Why can you not just use the canonicalized URL to detect dupes? That is infinitely simpler than doing text analysis.View on HN