Link shorteners: the long and short of why you shouldn’t use them
gcs.civilservice.gov.uk
gcs.civilservice.gov.uk
ourapp.example.com/auth/reset?user=blah&token=1af17e73721dbe0c40011b82ed4bb1a7dbe3ce29eae4997c84600287f8866673d05fdaa1aa841a5a
and they they figured, oh man, those links are unsightly for email, we'd best turn that into something like: ourapp.example.com/s/xO8pR
That looks way cleaner in an email.Wow. And makes it easier to brute-force (Which I think you're insinuating). If the links have an auto-expire of 10 minutes, is the risk sufficiently mitigated? Or am I missing something else?
You'd need to rate-limit the shortened URL endpoint as well or increase the number of characters. Without it, you could reset a user's password and brute force all shortened possibilities while entering their username. There'd be enough red flags to identify and stop this type of behavior, I think.
Even if it weren't, triggering a password reset for somebody and then fuzzing the url space ends up working pretty quick. I saw a few comments talking about rate limiting but since these are by definition unauthenticated requests, that's not going to help you either.
Short version (heh) is you cannot make this secure.
And, of course, tracking.
This. So much fun can be had by enumerating link shortener URLs. I've experimented with enumerating some services' URL schema. Most of the time the link pointed to innocuous things like Amazon affiliate links or whatnot. Sometimes you would find interesting content that made you go 'wow!', but that was very rare.
Not really. Usually password reset tokens are only valid for 10 or 15 minutes. With some basic rate limiting, you can stop a single actor from accessing more than one of those links in 15 minutes.
And even if they work around that, you just ask the user to verify their email address when they click on the link. Being able to enumerate the reset tokens and guess the right email address at the same time is highly unlikely.
Attacker knowing some existing user email will go to "forgot password" view and type in the e-mail for the user they plan to attack. Then after will start bruteforcing the token.
It is highly unlikely they had rate limiting because they had long tokens there for a reason and most frameworks like Laravel for example which provide similar forgot password feature won't by default rate limit those tokens or at least haven't in the past. I am not up to date with current version of Laravel and I think it may be using signed urls instead. Which would also be obviously terrible if shortened.
So the original team who built forgot pw didn't expect someone in the future to start shortening those urls, so it is unlikely they figured rate limiting to be necessary in this case.
It would require in most cases conscious decision making and effort to specifically rate limit token guesses, likely to be out of scope.
Catch all rate limiting by IP wouldn't work either because it would be arbitrary to use botnet to bruteforce.
But in the OP example the e-mail/user was already in the url so included with the shortened url. In this case hacker could just try random short urls until they hit something and due to redirection also immediately know the e-mail.
How?
Everything you said is true for the implementation that was listed, but my point was short URLs for password reset aren't always bad, if other mitigations are in place, which should be in place anyway (rate limiting requests for password reset URLs and requiring verification of the email address).
In this scenario, no. We're talking about an attacker who is iterating over the ID space of a URL shortener in order to find a password-reset link that had been URL-shortened.
If that link only contains a token and not the user's email address then the attacker doesn't know the email, and asking them to provide their email on the password reset page would at least make it marginally less bad to url-shorten links containing password reset tokens. (But it's still a bad idea)
But well, yeah in a scenario where you truly have short tokens and someone is not trying to create the token for the e-mail themselves, but just trying to find vulnerable links, then yes, in this case it would help, but that's non-sense, just have longer tokens and you are fine, why force user having to do extra steps, when you can just have longer tokens.
It's not just unattached performers who are the threat. People of every relations hip status and profession could be attacking.
Suppose you have 64-bit one-time identifiers for password reset links. With base-85 encoding, it's 11 characters, short enough to even type manually.
I suppose a password-reset link should expire within an hour. Scanning the entire 64-bit space in an hour in search of a working password-reset link is infeasible: rate-limiting will prevent it, and monitoring will warn about the attempt.
A feasible attack vector could be on the generation algorithm, but I suppose a good link shortener won't use a simple predictable RNG.
... and you lost your argument.
There's a reason why there's a variant of Base64 designed for URLs.
Anyway, I think that I'd be okay with a shortener like your example as long as it:
1. Required me to enter my user id again 2. Was only valid for 30 minutes
You can't do it for everything... for one thing, you don't gain search engine karma from the links. But it's often very useful.
Still, it can eliminate lots of unsightly cruft in a link. It can replace this: https://thisonecompany.com/productinfo/specs/?sku=sdfdf432&f...
IMHO, this is a display layer issue that only affects human eyes, and should be handled on display layer (html, email rendering...etc) -- just don't display the whole URL somehow. Machines won't have any problem processing that long link.
However, it really infuriates some vocal technical folks (e.g. https://www.androidpolice.com/2020/08/13/google-resumes-its-...). I think the compromise is good: hide the full URL by default, but have a setting or some affordance to show the full thing to people who do enjoy looking at it.
Edit: This is wrong and should be 30 bits of information not 320 bits for the shortened form. 64 values = 6 bits not 64 bits.
No, 64 values – or 6 bits. 6×5=30, not 320.
At reddit we had a link shortener (redd.it) that was automatic for every post, which was useful for posting on social media, especially twitter, when the limit was 140 charters. There are lots of other uses for internal link shorteners too, like just having nicer URLs for publishing or saying out loud.
But yes, the article is totally right about 3rd party link shorteners.
> You can request a short URL if you’re the GOV.UK lead or a managing editor in your organisation.
> Submit a request for a short URL at least 2 weeks before you need it. GDS might not be able to meet your deadline if we do not get the full 2 weeks notice.
They never did implement that, but it sounds like it might be a good general rule for many web sites that accept and display content from users. If you're concerned about the way long links appear you can abbreviate them on the screen (the way HN does).
An example in this paper cites the shortener used by Google Maps. The researchers were able to enumerate all the short links by brute force and join destinations from specific residential addresses. This is scary because now you've essentially created all points of interest that 1 person visits (originating from their home address).
Google's response was to expand their URL tokens from 5 characters to 12. The sparseness makes it uneconomical for someone to brute-force their way through. Microsoft OneDrive's response was... interesting.
Published articles meant to be accessed publicly seem like a case for the former. The idea is for those references to be found, and a search space which is both predictable and small is preferred. Here I tend to like schemes such as:
example.com/yyyy/mm/dd/nnnn.../<optional descriptive>
That is, for temporal data, explicitly code in the year, month, and day (and finer gradations of time if appropriate), then an item number (possibly sequenctial). The optional descriptive text might incluce author(s) and title(s).Dates aren't always required. Some well-known cases (comparatively) are Amazon's reliance on SCU, iBiblio's reliance on ISBN, and Worldcat's reliance on OCLC. (You can omit all other index elements on the URL to obtain the desired result.)
Sparse spaces tend to be for non-published / non-public entities and docucments. Google+ in particular had a 20--21 digit numeric userid (apparently used within Google as the account UUID). Even with some 3--4 billion registered profiles (the vast majority auto-created through Android device registrations), the space was sparse to a ration of trillions (and higher when interest was focused on the only the 100--300 million or so active accounts). This had a huge impact on the ability to crawl the space efficiently, as a brute-force search would have taken some time. Fortunately, Google provided sitemaps....
A related concept is James C. Scott's notion of legibility (from Seeing Like a State), and where it is and is not advantageous, and for whom.
... just create an S3 bucket with a short domain, configure it for static web hosting, and upload empty files which have the "Redirect" metadata property set to the destination URL. Voila!
You won't have analytics (maybe this can be configured via AWS, but I can't say) but you don't need a server either.
I want to eventually create a friendly control panel to create and delete shortcuts using React, AWS Lambda and Cognito... but I still haven't had time... and we only need to add a handful of short links per year. This can also be scripted and done quickly through the CLI.
Heavy “Dropbox is just cvs mounted over ssh, easy!” vibes over here.
And the admin panel provides a simple way to edit the KV database, so you don't have to write a db editor.
But, yes, not free if you exceed 100k requests/day. $5 per 10 million requests beyond that.
The idea of fronting it with the actually free regular cache is interesting. There is an API to control that "regular cache", so you could probably control that from the side rather than chaining the proxies/domains.
https://www.outsidethebeltway.com/libya-the-internet-and-bit...
72,287,136,510 links scanned, 14,632,045,317 shortened URLs archived. If you are interested it's easy to run their docker image to help with this and other archival projects.
I run a link shortener site, and use it privately and don't publicly expose the API.
One thing I noticed regarding analytics, is that the click count is always skewed. When I post a shortened URL on Twitter, within seconds the click count is always `>10` views. After further investigation, it seems there are automated bots that scoop up URLs the very second they are posted.
Also Twitter runs little microbrowsers that scan the page for metadata which helps them create a 'preview' of the link.
After looking at the useragents of some requests I'm seeing generic Firefox UAs which I can only assume are random surveillants (not bots) who habitually scan Twitter for interesting or anomalous content. We truly do live in a world where nothing is left `unseen` (by bots or actual humans).
Pro-tip: append the “+” character to any bitly link to show the target link without first visiting it.
Pro-tip2: consider browsing with JavaScript disabled by default. Enable it on a per-domain basis.
You're right. Short URLs are Shite urls.
I never click on links unless I know where it is going to lead me. Shortened links are one example. Even with an accompanying description, it raises red flags. Links to reputable image or video sharing sites without an accompanying description, is another example since you never know what is going to be on the other end.
[1] https://github.com/DandelionSprout/adfilt/discussions/163
[2] https://old.reddit.com/r/uBlockOrigin/comments/m5iecq/how_do...
Description: In a world dominated by bit.ly, ad.fly, and several thousand other malware cover-up tools, this list reduces the length of URLs in a much more legitimate and transparent manner. Essentially, it automatically removes unnecessary $/& values from the URLs, making them easier to copy from the URL bar and pasting elsewhere as links. Enjoy.
I've no idea who (even which party) initiated it, but it was just sort of suddenly awesome. Or maybe it just evolved rapidly under great (civil) leadership and was 'always' there just not so great.
Blog has lots of good stuff too, especially articles on accessibility often do well on HN, since that's something they 'obviously' need to worry about, and actually do & do it well.
Often in comments here too (Robin something is a username I recall) - not the sort of crusty 'what's an HN?', 'you can't do that because the Oracle database on our IBM mainframe doesn't support it' department I might've formerly imagined at all.
>> 'you can't do that because the Oracle database on our IBM mainframe doesn't support it'
Oh gawd that's literally my day job... :O[0] https://en.wikipedia.org/wiki/Government_Digital_Service
(And commenting on an article hosted at civilservice.gov.uk no less.)
DB2 on IBM z is quite well capable, I'd just like for there to be less artificial barriers between teams involved in the places I encountered it :/
(Funnily enough some of it could be blamed at bright-eyed "modernizers")
Sadly they are encountering resistance though. Some departments would rather spend £X bn contracts with HP, Fujitsu etc. so they can retain more control.
Also they used to publish amazing service status dashboards, showing how many transactions were published, error / success rates, etc. for every digital service.
Apparently these were all killed off recently with no replacement, and no good reason given.
Directgov 2010 and Beyond: Revolution not Evolution - https://www.gov.uk/government/publications/directgov-2010-an...
Google Analytics and similar are blocked by a large (and increasing) number of visitors. To my estimates, about 40-80% of a website's visitor will not be counted in Google Analytics (depending on website and audience). Some browsers now block those platforms without the need for any add-on too (like Edge in "Strict" privacy mode).
In short, GA is useless or soon will be.
So, yeah, I'm not sure how much longer they'll be a viable source of data.
Given that this is a mainly message for those communicating on behalf of gov.uk, I think the best they could do is host a URL shortener for use by government communicators. It's also good advice for businesses.
> If you’re adding campaign URLs to offline materials – like posters or leaflets – and don’t want to feature a long web link, I’ve got good news for you too. GDS provides the option to you to request a shortened version of a full GOV.UK URL.
I'm disappointed that they mentioned Google Analytics. People willingly using Twitter (or Instagram) is a thing, involuntary Google tracking is another.
The generated QR code had the URL rewritten to a short URL, and buried in some small print was a limit to how many times the URL could be “scanned” before you have to pay.
I guess these sorts of sites _really_ count on people missing this and spending thousands on print before realising.
Custom domain, of course, or the carriers wouldn't like it.
Now that I'm thinking about it, I should add bitly and related to my DNS blackhole...
IMO, This defeats the context of the article.
DDG "url expander" returns a number of these. I've been relying on the first result, https://urlex.org/ , for some months now, particularly as my router/firewall blocks most actual shorteners as spam vectors.
Note that if the shortened URL contains any specific private information, or would identify you specifically, you're still facing a risk. For shortened URLs found "in the wild", they're a useful tool.
I understand the argument to use a secure, formatted, link shortened. I don’t understand the argument to use Google Analytics (which captures way more data than I need in most cases) as the full and final solution to all of my analytics requirements.
Control your links, override slug names so they are readable, maintain private analytics, keep branded by running on your own domain.
It’s easy to set up and maintained by many of the people working on WordPress core. I recommend it.
You can give us feedback at https://github.com/alphagov/open-standards/issues/75
P.S. Service discontinued, but a lot of links are available. Bitly and Ow.ly support will be cool too.
You don't need to link shorten, because social media does this already. But also, shortened links are bad and unprofessional.
Don't worry - GA will do analytics. But also, watch out for privacy.
> `How to request a short URL*
> Submit a new feature request using the support form.
> You’ll need to tell us:
> - the reason you need a short URL
> - the content or page the short URL will link to
> - how your short URL will be used in marketing and promotion
> - the channels you will be using, the number of users who will be targeted
> - what the main message will be in your marketing and communications
> - how many government departments or organisations will promote the short URL
Edit: I know it's required by ICANN (I read the emails) it's the "click here" action that bothers me and perpetuates dangerous behavior.
Probably. It's been a while since I worked in that industry, but ICANN has always been pretty picky about the exact contents of emails.
Changing the user workflow in the way you're describing is out of the question. The entire purpose of clicking a link in the email is to confirm that the email was received at the domain's WHOIS contact address. Allowing a user to log in and click "confirm" without clicking a link in the email wouldn't confirm that.
This is my issue. I've spent years telling clients to not click any links in an email from your bank/insurance etc. but yet Network Solutions and others are still putting a big fat "Click here" button in the email.