Hacker News Reader

Comment by CivBase | Hacker News Reader

I wish that tools which remove unnecessary information from URLs were more popular than traditional URL "shorteners".

For example, if I go to some random product page on Amazon, the link looks something like this:

  https://www.amazon.com/Really-Long-Product-Name/dp/PRODUCTID/?_encoding=UTF8&pd_rd_w=garbage&pf_rd_p=long-garbage-value-that-doesnt-matter-to-the-user&pf_rd_r=MORELONGGARBAGE&pd_rd_r=another-really-long-garbage-value&pd_rd_wg=garbage&ref_=even_more_garbage&th=1

That link looks ridiculous and isn't suited for sharing in many situations. For most people, this is the link they really want:

  https://www.amazon.com/dp/PRODUCTID/

I know there are tools out there that can shorten URLs like that. I used to have a browser extension which did exactly that. It doesn't work for everything, but at least your link wont die because a third-party service shut down or messed up.

View on HN

It's kind of tricky to do in general case, e.g. even hackernews is keeping meaningful semantic information in id= query parameter.

Because of that it ultimately needs to a site-specific database/algorithm, perhaps with a fallback to the default behaviour like simply cleaning up the most common garbage like (_encoding/usg/etc). I suspect it's possible to use some sort of machine learning to guess the meaningful parts of the URL path/query/fragments, but even for that we need some human curation for the training set. I wish we could collaborate on a shared database/library for that, have sketched some ideas/applications/prior art here: https://beepb00p.xyz/exobrain/projects/cannon.html

I started thinking about it since I have a similar problem in Promnesia (https://github.com/karlicoss/promnesia#readme), a knowledge management tool I'm working on. Ideally I want to normalise URLS, so they address the exact bit of information, and nothing more.

It doesn't cover everyone, but you should know about rel="canonical" (https://datatracker.ietf.org/doc/html/rfc6596). For example, Amazon helpfully hints that this messy Kindle Paperwhite link (https://www.amazon.com/Kindle-Paperwhite-Signature-Essential...) is actually https://www.amazon.com/Kindle-Paperwhite-Signature-Essential....

Additionally, CleanURLs to the rescue! https://github.com/ClearURLs

Yeah, sadly, to get the canonical attribute, you need to fetch the URL first (which is slow and wasteful). Also sometimes canonical would still be different on the desktop and mobile version of the site, so it still has to be normalised after that

I don't even think you need a machine learning algorithm. I follow a simple process all the time to achieve this.

  load the page with the original URL
  for each part of the URL:
    remove that part of the URL
    load the page with the modified URL
    if the page rendered differently:
      put that part back

You'd need to incorporate an ad blocker, otherwise changing ads on each reload could screw it up. Of course, you'd also probably want to hard-code in logic for popular websites like Amazon to avoid wasting time with reloads.

And I know there are limitations to the concept. It wont work for pages which require authentication or any kind of session data. It wont work for pages which are intentionally dynamic. It wont work for sites which cannot be accessed by the service. It wont be useful for pages who simply have long URLs. But it probably covers the many common use cases for a URL shortener, and you could always fall back on traditional shortening methods (redirecting) when it doesn't work.

Oh nice, I like it!

So it basically automates detecting useful bits for a particular URL, but it's kind of time consuming and flaky. It could be very helpful to populate the 'rules' database though, and then this database could be shared with other people so they don't have to scrape.

I guess when I said ML (or preferably some fuzzy algorithm/heuristic), I was referring to generifying rules so they also work on the sites not in the rules database. If humans can detect garbage in the URL looking at a few examples, the computer can too :)

FYI: https://amzn.com/dp/PRODUCTID is official, to shave off 2 more bytes.

Affiliate links used to catch some heat on HN but https://amzn.to seems to be showing up in more and more comments without too much blowback. It wasn't mentioned much that affiliates could view everything people bought while under their cookie, but I'm not sure if that's still possible.

Reply on news.ycombinator.com