Attackers exploit flaw in web’s security to steal $2M in cryptocurrency
freedom-to-tinker.com
freedom-to-tinker.com
But guess what? The Bitcoin network itself is _also_ vulnerable to BGP hijacking, but the attack vectors are much more complex and so we haven't seen many of these attacks yet. The literature is really interesting though, e.g. Apostolaki 2017 [0] and Tran 2020 [1].
[0] https://btc-hijack.ethz.ch/files/btc_hijack.pdf
[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=915...
We discussed this in an earlier blog post on freedom to tinker: https://freedom-to-tinker.com/2018/04/11/routing-attacks-on-....
A more technical description of these attacks is available in our Communications of the ACM article: https://cacm.acm.org/magazines/2021/6/252822-securing-intern...
BGP hijack during domain validation to get a badly issued ssl cert, then use that for MITM. I guess it will happen again.
But during that time, wouldn't they have also received HTTPS traffic from real users trying to load developers.kakao.com? I understand BGP hijacks can be somewhat localized, but still. The attackers presumably are getting real user HTTPS traffic for a domain for which they don't yet have a valid certificate. Which means those requests would fail to negotiate TLS properly, or the attackers could just not run anything on [IP address they hijacked]:443, which would make the requests for the JS file on developers.kakao.com timeout.
My question is, wouldn't real users whose traffic was also hijacked during that window before the TLS cert was issued notice the web application not working? Wouldn't a rise in incoming support requests or something indicate that something was breaking?
Our work with Let's Encrypt demonstrated the feasibility of validating domains from multiple vantage points on the Internet: https://www.usenix.org/conference/usenixsecurity21/presentat...
We hope and advocate for universal adoption of this approach via the CA/Browser forum (since the internet is only as strong as its weakest link).
In the case of the KLAYswap attack, based on public routing data, it is highly unlikely the adversary forwarded any traffic back to Kakao. As mentioned in the comments, certificate issuance only takes seconds, so it would not have been a very noticeable outage. Also, the adversary did not seem to mind causing connectivity issues given that they actually checked the referer HTTP header to only serve their malicious JS file to people that were downloading it from KLAYswap.
Best, Henry
How was the this client-side JavaScript being used as part of the security of the transaction? Shouldn't the transaction have been recorded as part of an atomic database transaction, which should only be possible on the correct server?
I don't know much about this specific third party javascript and it's possible that tighter CSP policies could have mitigated some of the attack vectors. However, once you can MITM, it's really game over.
Before we blame usage of the js CDN, they could have performed the TLS attack on KLAYswap instead and then just reverse proxied while strategically rewriting data. It's possible that with this specific setup that the js CDN was the lowest hanging fruit.
This is such an fundamental and structural attack (BGP poisoning to screw with certificate ownership validation) that I find the comment by Phill Hallam-Baker incredulous. Perhaps there is a hobby horse against automatic certificate exchange (there is a mention on EV certs, which were such an obvious scam) or it's just a middlebrow rant about "javascript dependent pages".
There are many attacks possible with this method and plenty that will work on the simplest request/response Web 1.0 architectures.
i.e. make the same POST to their api, using the cookie/auth as the original user would, to transfer the files. Its just an ajax call.
Outstanding research! I particularly enjoyed how the presenter in the video shut up the arrogant person from the audience.
Sure you might know somewhere the internet was broken, but you wouldn't know what caused it. It's a downstream issue from you so it's hard to deal with beyond filing a ticket with with some service provider.
Rather the conclusion should've been that the browser gives the user a warning when attempting to submit information to a website that doesn't have an EV security certificate.
Or at the very least allow websites to use HSTS to prevent downgrading an EV certificate.
Sometimes, the alternative to companies paying hundreds of dollars per year is customers losing millions.
Not every building needs to implement the security measures of Fort Knox. That doesn't mean that the security measures in-place at Fort Knox shouldn't exist.
It's a weird combination of inspiring and humbling at the same time. Inspiring to think that this massive ecosystem of interconnected computers was actually made by real people, and humbling because yeah, I should get off the damn grass.
What I often find most valuable from these kinds of comments is the fact that these risks were acknowledged and either accounted for in the original specs, even if in a flawed manner, or explicitly discounted as being out of scope.
What I take away from the comment on the article is that from a security perspective "we're not there yet" even if the cryptocurrency technology is useful, widely adopted, and frankly not going away.
I don't think his issue is that cryptocurrency exists, or is being used widely on the world-wide web, but rather that underlying technologies (DNS/HTTPS/PKI) are being taken for granted and used in a manner that they weren't originally designed for.
Cryptocurrency, Bitcoin, is expensive bits issued by a non-trusted third party.
I don’t see how you can be bullish on the first without being bullish on the second, yet I can see how being bullish on the second can mean bearish on the first. Solutions involving no TTP are better than solutions involving TTP.
I like PHB and he has done great things in the field of cryptography. His belief in centralization of public key systems for security runs counter to the cypherpunks who want to decentralize everything despite the costs.
---
*TTP Transactions*
When transactions are brokered through a TTP such as via swiping a credit card at Walmart. The transaction is then brokered by Visa, Mastercard, et. al. who retains the ability to reconcile various types of disputes & fraudulent transactions after the fact with minimal loss of money. At any point the central authority can technically reverse charges without the consent of either originating party.
My non-expert, "educated layman's" understanding is secure online digital communication was designed with the express intent of being used in this kind of scenario. You have a number of transactions of which some non-zero quantity are fradulent, relying on the affected parties to initiate some kind of arbitrage via the trusted third party. Prior to crypto, very few (if any) transactions performed online were irreversible economic transactions requiring *full trust* in the security of the communication channel to mitigate fraudulent transactions. Which brings me to what I believe to be his primary point of contention, putting gripes about specific decisions and technologies aside...
---
*Non-TTP Transactions*
Crypto currencies that require zero-trust in a third-party have no(?) mechanism with which to mitigate fraud.
That means your entire chain of communication, from physical layer all the way up to end-user, must be hardened against forms of irrecoverable tampering, that would otherwise be a recoverable nuisance in a TTP model.
As one of the engineers who designed & put into place a large part of the security around the communication channel being used PHB appears to be taking offense to the assertion that this exploit was due to a flaw in "the web's" security.
Simply put, it was never designed to be the first and final layer of defense against fraudulent activity, and if we want crypto currency to be 'the future' we should walk in the path of our predecessors and design a communication stack that explicitly addresses the concerns of being used for non-TTP transactions.
We need to harden the stack at all layers, as many components were not explicitly defined to protect under the zero trust model, especially with the exploitable financial incentives that could not have existed before. Thank you, I feel your insight has helped strengthen my understanding too.
> The system we built in the 90s would have protected against this attack.
Is "the system" the human to human EV certificate process? There were so many obvious and awful social and technical engineering holes in that process that I find it hard to believe anyone would defend it?
Certificate expiration was always a danger. There was a single technical contact that could be spoofed or just walk away with the private key. The wider web audience never noticed the EV annotation, and basic human factors tells you they never would. Heck, we can't get people to notice HTTP downgrade attacks. The only protection is TLS everywhere - which never would be possible with slow and expensive authorization methods.
(I suppose it was good business for Verisign though...)
Edit.
Seems that some preventative steps are....
- have all your javascript assets hosted under your domain.
- have a CAA record see https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...
- If you use a third party to host JS then THEY need to have CAA configured or you need to use sub resource integrity.
In this case it seems like the attackers targeted an SDK. Subresource integrity would have helped here.
https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
In any case they could've hijacked the IP for the authorative DNS server, but that would at least add some complexity.
Also, this assumes their CA actually did their due diligence and the hackers didn't just fool them into reissuing the certificate to them.
So if your site pulls in js from another site without sub resource integrity, and the other site doesn't have CAA configured you are vulnerable.
Personally, I think anyone who has a heightened attack risk ought to contemplate a CA that does some form of more thorough validation.
Sub resource integrity wouldn't help if they could have re-routed requests from klayswap.com
The best you can probably do is pay for a TLS certificate signed by a CA that only signs certificate requests after actual, human verification, and pin that in a CAA record. This should prevent other certificate authorities from handing out a domain-validated TLS certificate.
If you're going for the monitoring route, keep the amount of domains your site connects to to a minimum. Self-host your fonts, frameworks, etc, so you only need to monitor your own host. If the developers of the attacked site had hosted their own JS, they might have noticed that something was up when the attack took place, because they can't know for sure if a route change for an external company like Kakao is intentional or a mistake.
The core of the problem is that advertisements by routers that have no business announcing specific routes like these are taken at face value. Certain ISPs that do not require RPKI or other security mechanisms will happily route traffic headed for your IP addresses to some router on the other side of the globe.
Web technologies like DANE used to be popular for this purpose (using DNS to distribute the certificate of your website, with DNSSEC validating the DNS records) but it has seen little actual implementation so far. HTTP key pinning (HPKP) was implemented in some browsers for a while, but its potential for holding websites hostage and general unfriendliness towards bigger websites caused it to be phased out.
If you have apps that operate your service, you can pin your TLS certificate in those and that should protect you against such hijacks.
> In multiple vantage point verification, a CA performs domain control validation from many vantage points spread throughout the Internet instead of a single vantage point that can easily be affected by a BGP attack. As we measured in our 2021 USENIX Security paper, this is effective because many BGP attacks are localized to only a part of the Internet, so it becomes significantly less likely that an adversary will hijack all of a CAs diverse vantage points (compared to traditional domain control validation). We have worked with Let’s Encrypt, the world’s largest web PKI CA, to fully deploy multiple vantage point validation, and every certificate they sign is validated using this technology (over a billion since the deployment in Feb 2020). Cloudflare also has developed a deployment as well, which is available for other interested CAs.
> But multiple vantage point validation at just a single CA is still not enough. The Internet is only as strong as its weakest link. Currently, Let’s Encrypt is the only certificate authority using multiple vantage point validation and an adversary can, for many domains, pick which CA to use in an attack. To prevent this, we advocate for universal adoption through the CA/Browser Forum (the governing body for CAs).
That defense alone is still not perfect ("some BGP attacks can still fool all of a CA’s vantage points"), but that's the state of the art.
You don't get the self sovereignty without the responsibility.
You hope the FBI or some other law enforcement/criminal investigators can track down and prove the guilt of the thieves.
Jumping to the question of "who do I sue" implies that you want to get money out of someone not directly involved in the crime. In the example of gold in a safe, the question might be "can I sue the safe company to try to recoup some of my loss, even though it wasn't their fault that my opsec was shit?".
If there exists no explicit legal agreement that was violated (e.g. the bank that promised to secure your gold was robbed) then you're just being an overly litigious asshole acting in a way that will make our entire society weaker and more vulnerable.
What are you talking about? This is a standard theft/robbery case, the police absolutely file, investigate, and solve such cases regularly.
Am I the only one that yelled "NO" at their monitor when reading this snippet?
It is usually not good practice to load js from 3rd-party domains into your app. Just grab the file and serve it yourself.
For things like payments, I would hope that most of it is server-side, and that js doesn't play a critical role in it at all.
The PKI is great, but there are times when you really don't want to have to trust a third party...
1 https://letsencrypt.org/2020/02/19/multi-perspective-validat...
The only way to survive a BGP hijack intact is also a deal breaker for today’s typical common web browser.
Web browsers do not bother with DNSSEC (which remains quite very relevant despite what the article doesn’t says.
It is true that this particular BGP hijack doesn’t change your endpoints’ DNS records (such as A, AAAA, PTR, NS, MX). But this attack did not attempt to fake the DNS CAA at all, which would have protect web browsers; Faking CAA record was not needed then, but it will be leverage next. So DNSSEC, being secured as they are now however is not culpable there nor would they ever be. It is only the DNSSEC-unprotected CAA record that will be the next target.
The Culpability is the malicious PKI substitution which occurred during that event is the fault of web browser, but it is NOT the fault of the TLS protocol, nor HTTP; web browser is guilty. It is the design of using TLS function calls by the application. TLS API, while makes heavy use of PKI certificates, leaves out PKI management to the “other people’s problem” category. TLS protocol and its API is not the problem, they still work as designed.
PKI management is the problem. By direct user experience, web browser is also the problem.
However … your data still can survive a BGP hijack (DNS or not) as intact and securely so if your use case for B2B calls for a secured endpoint using HTTPS/mTLS that has the following aspects:
- use a pair of server and client PKI and its public key stored on opposite ends for use with mTLS (mutual two-way server/client TLS)
- DNSSEC resolver on both ends configured in cacheless authoritative-only lookup mode (no forwarder) with the 13 Root Server key configured/compiled in.
- Deploy the DNS CAA record for your PKI and ensure that each of the chained PKI uses CAA and ensure that all that needed CAA are secured that under DNSSEC
- your endpoint app properly use TLS/SSL function calls AND verified the PKI chains but not before verifying DNSSEC-only DNS queries needed for chain validation.
It remains arguable that given above, one can still use a 3rd party intermediate CA to sign your intermediate CA in which to deploy your needed PKIs for your endpoints. I am now asserting that if you separate your server PKI and client PKI to each of two different vendors, you should get a pretty good security provided that you figured out your own PKI distribution securely and that your own intermediate CA PKI is also imprinted into your app’s own mTLS.
That summary should help your data survive a BGP hijack in your “B2B”pipeline.
A bit safer than IPSec.
Thank you.
This DNSSEC-only DNS caching is not a common thing to find in most desktops because it impacts many web sites that do not use DNSSEC (looking at you, Google.com).
Of course, this is assuming that your cryptowallet provider is ALSO using DNSSEC properly.
Sadly, none of today’s web browsers (including all of the DNSSEC Validator web browser extensions, that I could find) perform any detection for bad/unsigned DNS queries , much less do any active defensive blocking.
I blame the web browsers’ developers for this disabling of these DNSSEC validation capabilities through their poorly-planned but overly restrictive efforts in deploying newer browser extension APIs.
Those tested DNSSEC Validator browser extensions USED to be working but only back then and ONLY in passive detect mode; nowadays, they’re just a dead extension garnering nothing but downvotes).
Your only practical day-to-day recourse to all that jazz is to establish a windowed VM that does this DNSSEC-only DNS caching and any choice of a major web browser for the bestly secured crypto-exchanges. QubesOS is one such excellent option but VMware or VirtualBox would do just fine in a pinch.
Of course, TLS, HTTPS, JavaScript and TCP attack vectors are another whole ball of wax.
This is why we can’t have nice things (such as TLS Client Side (or mutual TLS) easily enforced).
But who does that nowadays? I still do for my mutual TLS APIs.
It’s only if the client public key (which is stored only in the web server could ALSO get hijack then all that would be … for naught (to the hacker’s gain, that is).
Not necessarily a properly configured Mutual-TLS non-browser API handler, but the article is talking about what most people use which is a web browser.
TLS permits client (initiator of the session) and server (target of the session) to have certificates. The purpose of the client certificate is that the server may authenticate the client. The purpose of the server certificate is that the client may authenticate the server. Hence: mutual authentication.
If the server is replaced by a malicious implementation that has the private key associated with a genuine certificate, the client will validate the server as authentic. This happens first in the TLS handshake.
The malicious server will then happily receive the client certificate and accept it without any chain validation, because it doesn't really care whether the client is authentic or not.
To repeat the point: the use of client certificates does not and cannot protect the client.
But that defense would only stand by requiring a prior imprint of TLS signatures at both endpoints which no web browser would be able to do during a mutual TLS unless that web browser also starts to takes not only the mutual-thingie client TLS cert but to remember the TLS server public key cert as well beforehanded; not a pretty easy thing to set up by an average web user if the web browser choose to support this.
But you can do this strong protection of dual public key exchange for your own set of endpoints and no BGP hijack can get to it unless both-side’s private keys are compromised; it is just that web browser cannot bother with the prior fingerprinting of TLS server due to a variety of reasons (namely expiration, revocation, domain churn, …).
This new big data collection of TLS server certa surely can only be rudimentary protection and certainly not against a timing attack, no?
My assertion now is that it is the current design of the TLS handling by web browsers is purposely crippled for ease of use purposes. TLS protocol, however, remains robust against such a MitM at IP reroute-level if verification of TLS certs are done both ways and on BOTH sides; kinda like IPSec, right?
This is fine in a closed ecosystem, where you can also compel your trust roots into the clients and have a mechanism where you can push out updates to endpoint/public key mappings. This is why you still see HPKP for mobile apps.
If DANE were actually a thing then it would also be an option for more open internet use cases; but the reality is that it too comes with its own security and operational problems, and essentially no TLS implementation support as a result of the chicken-and-egg problem.
In any case, this is all outside of TLS, which only generally assumes PKIX.
Also back to mTLS (and a bit of DANE).
You left me wondering how to hardened against a compromised intermediate CA if I were to start to rely on my CA provider and their ecosystem to supply with a secured CA cert for my ease-of-management so that we can do all the things we need to do; but only for one direction of the mTLS.
And I am not even talking about the incomplete browser mTLS handling found in a typical web browser, but a well-designed mTLS server/client app.
There is a sharp difference between HTTP-based mTLS vs. Browser-based mTLS. Browser-based mTLS is sorely incomplete and does not block well if verification of CA chaining failed.
Fortunately, for me, I think, if just one half of non-Browser mTLS exchange are under the domain of a private Root CA, there should be that small measure of the thin layer of security remaining … against a BGP hijack as long as the fixation and memorization of two mutual (public and private) Root CAs is done at software level (of which such properly secured mTLS are still not found in all web browsers but hopefully only in our non-web REST/HTTP API software).
We say “non-web” to be anything not useable by a web browser.
And we often hear new intern at test department complain loudly why they could not even use a web browser to exercise against our non-web REST/HTTP/mTLS APIs. We don’t even let them use curl/wget for their I&T stage (only within their unit tests). And even then curl/wget can’t get to our corner test cases.
At this point, I really hope we don’t need to consider DANE for our scenario.
Regardless of whether our own intermediate CA (signed by a public Root CA) that we control gets compromised (via BGP hijack), it’s a properly deployed non-web mTLS and our private Root CA covers the client-side of mTLS, by design we should be safe against such a BGP hijack.
Yeah, Private Root CA for the client-side mTLS and a fully chain-validation deployment of mTLS mechanism, we should be fine there.
I am quite sure that someone looked into this full deployment mTLS for web browser usage (as my Mozilla bug report is still open on this for years).
I should point out that BGP attacks can also target the CAA records themselves since the DNS ecosystem is itself insecure. This is why such attacks are not easy to defend against, and require holistic improvements across internet routing, PKI, and web security practices.....
To avert a DNSSEC-secured DNS CAA record from being tampered under BGP hijack, both endpoints must deploy their own cacheless DNSSEC authoritative-only resolvers before commencing mTLS endpoint communication. (Like a web browser would even do that /s). This places the trust of DNSSEC to right at the 13 DNS Root Servers whose private key are stored in a HSM key vault. So, hijacking a DNS infrastructure to tamper the CAA record is only possible because 1) web browser don’t bother with DNSSEC 2) domain owner don’t bother with DNSSEC protection.
But even with insecure DNS, I assert that a properly designed REST/HTTPS/“mTLS” API (but would then be totally unusable by web browsers) would still not be intercepted under BGP hijack scenario so that is still an option for any B2B scenario.
It was IP (at the routing layer) that was subverted; i.e. the IP addresses resolved from DNS remained the same but the traffic to them was redirected.
Was focused on the latter distinctions between:
- incomplete mTLS and lack of DNSSEC implementation (found in all web browsers)
- proper two-way mTLS with PKI validation of CA chain whose DNS CAA record is verified as properly protected by DNSSEC.
The latter distinction gets your data through a BGP hijack safely from any MitM; former one, not so much.
There are ways to protect against such attacks (RPKI for BGP routers, DNSSEC + DANE or HPKP for servers) but none of those aren't available in most modern browsers.
I'm a strong proponent of DNSSEC, but it can't solve _all_ problems. Only with DANE would it benefit the situation.
Key thing is thou that the authoritative DNS server must also reside on the web server.
And gotta turn that DNS cache off or keep TTL short.
Nothing to do with DNS which translates names to IPs
The IP addresses themselves were hijacked. DNS has no authority over IP addresses so DNSSEC would be pointless.
Often times, heavy non-commerce sites skimp on DNSSEC (such as google.com)
But why bother when web browser don’t even use DNSSEC itself? Because you can still send data securely under BGP hijack between your two endpoints with using DNSSEC-protected CAA record and a properly-CA-verified mutual TLS (mTLS).