Wikileaks moves to Amazon's cloud to evade massive DDoS | Hacker News Reader

Wikileaks moves to Amazon's cloud to evade massive DDoS

arstechnica.com

88 pointsevo_967 comments

Doesn't hosting with Amazon create a money trail? If the US were to prosecute them, they could subpoena Amazon, and find more members of the organization.

Assange is knowingly putting himself in the line of fire, but isn't there a goal to make sure other members of wikileaks stay anonymous?

Why should other members be anonymous? Isn't that hypocritical since Wikileaks is all about transparency?

It's also all about protecting whistle blowers. They've put a lot of thought into ways of covering the tracks of people who come to them to keep them safe - I don't think it's hypocritical. They're concerned for their safety, not their politics.

Ironically, Clinton expressed similar concerns about Wikileaks exposing people who speak privately to diplomats.

They asked the US gov to redact any sensitive sources. They didn't get any help.

http://www.salon.com/news/opinion/glenn_greenwald/2010/08/20...

Possibly because that would tell Wikileaks what they thought was sensitive.

I'm sure there are ways around that. Give Assange a boatload of prepaid credit cards, have him pay Amazon, something like that.

Moving to an elastic infrastructure in the face of an attack that aims to increase usage, seems like a recipe for a huge bill.. The wikileaks guys are obviously uber smart and have thought through the pros and cons of the move, but it seems an unusual choice..

As it's mostly static content, they're likely just using Amazon as a CDN, which isn't very expensive at all.

Makes perfect sense actually. It'll be interesting to see what Amazon has to say about this.

Using CloudFront (Amazon's CDN) does promise to solve their immediate DDoS problem.

I don't know much about wikileaks as an organisation but I expect most of their money comes as donations via the web site.

So, clearly having wikileaks.org up at high traffic times is important.

A competent hosting company - with a sufficiently fat upstream pipe - should be able to handle it. Amazon has the scale to absorb this attack.

I'm not familiar with Amazon's content policies, but I couldn't see Amazon wanting this stuff hosted on their servers.

It would depend on how Wikileaks designed their AWS infrastructure. Amazon's policies are specific to the territory in which they operate. They obey the local laws for the data they will store on their hardware and will only comply with local laws. For example, if you provision your services in the EU-Ireland region, it is governed by the laws of the EU, not by the United States. That being said, I don't know the specifics of the laws with regards to each of the different AWS regions.

> the laws of the EU

We are, sadly, not quite that far along yet.

Sadly? The things that come out of the EU are at least as stupid as local laws, especially wrt technology.

The US has a massive commercial advantage because its laws, while not better per se, are at least uniform.

Also, European national states are mostly too weak to have any influence in the global problems of the 21st century. A smaller share of a larger (power) pie would still be an improvement.

Economically, it would be an improvement. But look at laws like the data retention law that forces ISPs to keep access logs on all their users. The EU is also highly undemocratic, or at least very indirectly democratic.

First install a working democratic process, then get more power. Not the other way around "lets give them insane power and then they will surely be nice to us and give us a good democratic process" as many people seem to want.

Also I'm not even sure that if there were a good democracy that I'd want to give e.g. Italians the power to vote on what happens to me, given that they elect and keep electing Berlusconi.

Meh, the Netherlands went further than the EU requirements on data retention. Yes, that's stupid, but...

And yes, creating a "EU government" is almost certainly even harder than it appears. The current system seems to combine the speed of a multi-country democracy with the legitimacy of a multi-country oligarchy (of elected ministers, but still).

The EU issues Regulations, which have immediate effect in member states, and Directives, which member states are required to implement in national law, so it's not too far wrong to talk about EU law. http://en.wikipedia.org/wiki/European_Union_law

Caveat: IANAL (but my wife is).

Yeah, I know. You should also mention the European Court of Human Rights, whose decisions override national courts (of course, the decision is usually "this case not is our job, go with whatever the national court told you".)

Technically you are goverened by the law of Ireland, not the EU (per se).

However someone else pointed out they are hosted in us-west

Amazon is fine with it, according to the WSJ.

http://blogs.wsj.com/digits/2010/11/29/wikileaks-using-amazo...

"Amazon and WikiLeaks did not return requests for comment."

Why not? The New York Times and many other newspapers published the same data, and surely there's no problem hosting the NYT, so why is Wikileaks special in this regard?

I am curious who wants to attack Wikileaks and for what purpose. I think it's unlikely this is the USG because of the sheer pointlessness of it, plus the fallout that would occur if it were discovered. The likeliest major player I can think of is China, but I'm not sure I have a reason for that belief other than that they are the bogeyman du jour. An alternate possibility is that this is just some cracker flexing his muscle, or showing a potential client what he can do.

Shear pointlessness has not been preventing thr USG from doing much of anything in the past decade or so.

Or did you miss the past few years of security theatre and unwinnable wars?

Are you actually trying to argue with me, or just taking a potshot at the USG? If the latter, please use someone else's comment as a platform and not mine. If the former . . .

Sheer pointlessness is meant to encompass several reasons I believe the USG is an unlikely culprit.

First, there's the risk of significant consequences if the USG's involvement comes to public light. There's another election in two years, so even the PR damage could cost them. Then there's the matter of this kind of action being illegal. Maybe no prosecution would occur, but maybe it would.

Second, unlike your security theater example, there is literally zero chance of this doing any good for anyone. Every person who is even the least bit interested could obtain the documents with only minor effort considering how widely they have been distributed. Major news organizations have already collated them for the masses.

On the other hand, "security theater" and "unwinnable wars" are only pointless in a debatable sense. There are obviously a large number of people who believe that airport security measures are having some positive effect. Similarly, by some metrics (not mine), the war in Iraq has been successful and the war in Afghanistan is heading in that direction too. I don't doubt in hindsight that they will prove to be mistakes, but that is not the same as their having a absolutely known "pointlessness" value now. So I think that objection of yours is incorrect as well.

Also, you've only pointed out two pointless things the USG has done in the past decade. That's not the same as proving that the USG is equally inclined to do pointless things as pointful ones. In fact, it may be that the USG discards pointless courses of action at a much higher rate than pointful ones, but that certain pointless actions have been enacted nonetheless. In this case, it would still be predictive of the US not being involved that the Wikileaks DOS is pointless.

Finally, I'm not aware that it is the USG's standard infosec policy to maintain batteries of compromised civilian computers with which to perform cyber warfare. Perhaps it is, but that would be quite a discovery in and of itself.

So, there are a lot of problems with your line of reasoning. I could be wrong, but I think the odds are something like 80-20 that I am right and this is not the USG's doing.

Wired said some random guy did it, not a government: http://www.wired.com/threatlevel/2010/11/wikileaks-attack/

Note to other members: It's not OK to downmod someone for an opinion, especially not below zero.

It looks like it's hosted in the US (ec2-184-72-37-90.us-west-1.compute.amazonaws.com). Also, the front end proxy is doing some heavy filtering to weed out the cheap hit-and-run nodes participating in the DDoS but still accepts legitimate browser-based requests (persistent). Notice how a Reset (R) is sent right away on the first try:

08:57:41.211436 IP managed.unixy.net.49467 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: S 1398247905:1398247905(0) win 5840 <mss 1460,sackOK,timestamp 2031832550 0,nop,wscale 7>

08:57:41.264403 IP ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http > managed.unixy.net.49467: S 1288073904:1288073904(0) ack 1398247906 win 16384 <mss 1460>

08:57:41.264424 IP managed.unixy.net.49467 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: . ack 1 win 5840

08:57:41.318642 IP ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http > managed.unixy.net.49467: R 1288073905:1288073905(0) win 16384

The Reset packet sent from the EC2 node to the initiating node is a probe to identified non-existent nodes (spoofed). Notice in the above handshake that the initiating node didn't send a packet-response to the Reset. On the second consecutive attempt though all appears well (because the EC2 node added the initiating node to the ACL).

08:57:42.961708 IP managed.unixy.net.49468 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: S 1394481180:1394481180(0) win 5840 <mss 1460,sackOK,timestamp 2031832989 0,nop,wscale 7>

08:57:43.016547 IP ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http > managed.unixy.net.49468: S 1406181195:1406181195(0) ack 1394481181 win 5792 <mss 1460,sackOK,timestamp 26523835 2031832989,nop,wscale 3> 08:57:43.016564 IP managed.unixy.net.49468 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: . ack 1 win 46 <nop,nop,timestamp 2031833002 26523835>

08:57:51.100914 IP managed.unixy.net.49468 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: P 1:18(17) ack 1 win 46 <nop,nop,timestamp 2031835023 26523835>

08:57:51.180674 IP ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http > managed.unixy.net.49468: . ack 18 win 724 <nop,nop,timestamp 26525876 2031835023> 08:57:56.206546 IP managed.unixy.net.49468 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: P 18:39(21) ack 1 win 46 <nop,nop,timestamp 2031836300 26525876>

08:57:56.261630 IP ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http > managed.unixy.net.49468: . ack 39 win 724 <nop,nop,timestamp 26527146 2031836300>

08:57:56.678942 IP managed.unixy.net.49468 > ec2-184-72-37-90.us-west-1.compute.amazonaws.com.http: P 39:41(2) ack 1 win 46 <nop,nop,timestamp 2031836418 26527146>

But I wonder just how long they'll be able to evade the attack.

Regards

Surviving DoS attacks is not a new problem; I'm sure that Amazon has quite a bit of experience in that area. But thanks for the trace, quite interesting!

I wonder how well their Cloudfront CDN holds up for delivering static content in the face of a DDoS vs just using heavy static cachine with ngnix or apache on EC2.

Probably better, the CDN should be able to absorb much more traffic. (At 10Gbps, you need to worry about the attackers saturating your upstream pipe. Forget responding, just receiving that much traffic is nontrivial. EC2 nodes are probably networked using gigabit ethernet, so no matter how clever you get you can't solve this with everyone's favorite "single load balancer plus nginx web servers".)

If everything is edge-cached a local hubs (typically how a CDN works) then it should be pretty sturdy against a DDoS attack - after the first few waves of attacks the DDoS will just be hitting static content served up by the closest end-point to each individual ping to the server - the aggregate of the attack won't impact any of the dynamic servers if done right.

That being said, I wouldn't want to pay the bandwidth bill for the CDN :p

Check out the anti-DDoS article we wrote up a few months ago. It uses a constellation of Nginx nodes to absorb the DDoS. The constellation acts the same way as a CDN without the geoIP awareness.

The advantage of this approach is that instead of relying on one ISV's backbone, you can build the constellation across several ISVs to aggregate the bandwidth and packet processing power.

Regards

I would think this is interesting enough as a scale/anti-DDOS exercise to merit support from Amazon and other folks regardless of your political stance on wikileaks.

EC2 and Cloudfront are powerful technologies, here's hoping they're transparent and publish their solutions/tools.

I guess this explains why netcraft keeps showing their website as mostly down worldwide for the last day or so when it's actually performing extremely well.

Question: why does the RST packet identify non-existent nodes? Doesn't TCP sequence prevent a blind continuation of a http request? Is this just one type of syn flood protection?

Most likely, the initial Netcraft attempt to connect is met with a RST. Subsequent attempts are also met with the same RST. This is because the attempts are spaced out in time enough for the original ACL, permitting access, to be flushed.

According to the RFC, the RST does not get an ACK if the initiating node is "legitimate." So the "silence" or non-ACK is a good sign, which results in the initiating node being added to the ACL. You don't want plain TCP handling this because of half-open TCP handshakes which can exhaust kernel data structures (memory) and CPU (from having the kernel sift through a large data set).

Regards

Thank you for your insight. A bogus IP - one that no one is listening on - would also not ACK a RST, right?

Doing a little googling this process seems to detect an attack (from a valid ip) that has been programmed to ignore RST - presumably because some intermediate ISPs (like tier1 borders) will detect a DDOS and forge a RST to attempt to mitigate them. Much like the firewall configs that circulated to defeat sandvine RST throttling of bittorrent.

A bogus host will obviously not respond to the RST but the last router-hop that receives the RST will (per RFC/protocol specs). The response is destination unreachable via ICMP. The ICMP unreachable packet is cheap (non-persistent) and requires no up-keep from the last hop to the bogus host. Most importantly, the ICMP unreachable packet requires no upkeep from the filtering node in EC2.

Regards

thanks much!

ALL the more reason why Assange should stop prancing around like a f*ing queen and just release the entire trove in one big torrent. By dishing out the documents slowly (their own FAQ claims they'll release the docs over a period of months to maximize exposure), he's setting himself up for such attacks.

They can always release their analysis and interesting findings later.

I know DDOS is a difficult problem to solve, but I think using the cloud to out-scale your attacker doesn't solve the problem, it just increases the cost of it. The obvious solution for the bad guys (not gonna call them hackers, can't call them crackers can I?) is to use the cloud too. Generating an HTTP request is even cheaper than serving static content on a CDN.

> The obvious solution for the bad guys (not gonna call them hackers, can't call them crackers can I?) is to use the cloud too.

I'd say a botnet already qualifies as "the cloud". Why pay Amazon when you can get a million desktop machines doing it for free?

At college, we were already talking about "the cloud" in 1990. That's a long time before AWS was even a twinkle in anyone's eye.

Unfortunately, once the big guns of the media and those opportunists in government find out, Amazon will be forced to make a statement one way or another. Do they support this kind of thing or not? I can even imagine calls for a boycott from some sectors - not a great thing for Amazon at this time of year.

Everything that embiggens this story is good news for Wikileaks. And Amazon taking a stance would be something big.

According to this, Wikileaks is no longer available on Amazon either:

http://online.wsj.com/article/AP90b4520b2a9b455ea6e9d8d66fae...

I think after some days, WikiLeaks will publish documents using torrents. That way they can avoid (at least) DDoS.

They are already doing that. Not much good if people can't get to the .torrent file. (Yes, I know about DHT and that you can just mirror .torrent files. It's still inconvenient enough that most won't bother.)

This is very interesting. If Wikileaks does in fact become designated a terrorist organization by the US, then it seems Amazon will have to shut them down or run the risk of providing them "material aid".

The same would be true of any other cloud provider... are there any sizable cloud providers outside the US?

The overheated rhetoric spewing from some politicians notwithstanding, I can hope that no court of law will be persuaded that there can be a terrorist organization without weapons, indeed without violence of any kind.

(Even if Assange has committed some violent crime on his own -- and I am not saying he has -- this is clearly not the purpose of the organization.)

Does the government still need to go through courts when dealing with "terrorists"?

There is actually a bit of judicial review in the case of designating terrorist organizations, though I don't know how thorough. Here's Wikipedia on how terrorist organizations get designated, and how they can challenge it: http://en.wikipedia.org/wiki/U.S._State_Department_list_of_F...

Cloudsigma is based in Zurich, Switzerland and seems to be fairly big. They should be able to cope with this kind of attack:

http://www.cloudsigma.com/

If Wikileaks is a terrorist, then so too is the New York Times, and any other media outlet that has conveyed the same information Wikileaks released.

That is a logical fallacy.

Bingo. Bob Woodward publishes leaks like this all the time in his books. Some of it is TS stuff, too, IIRC.

Why don't they invest some of the donation money in P2P DNS and start hosting the site as torrent ? Many people would be happy to seed.

Because Bittorrent-powered P2P DNS is a weird buzzword, not yet really existing piece of technology. And, I'd guess, WikiLeaks probably want to publish (primarily) on the mass-accessible Internet, not at some obscure place where nobody except for crypto-geeks could access it.

They could try popularizing Freenet (which is already existing and is a fairly stable technology), but, again, I'd guess they probably have their hands full of other tasks already.

WikiLeaks hosts torrents. For example see the link "Click here to download full site in single archive" at the bottom of http://cablegate.wikileaks.org/. And, I believe, the content is already copied to (and being discussed at) Freenet and other similiar P2P networks.

If they used Freenet I can imagine the mainstream media going straight for the fact that Freenet is full of child porn. I don't think they need any more of that kind of publicity.

http://geo.flagfox.net/?search=wikileaks.org

http://e.businessinsider.com/nbl.2e8/TPa_4qqdbcEdB9HKA8022

there it is - i called it! http://news.ycombinator.com/item?id=1957189

on ec2? 10 bucks that site is gone by end of this week.

and i was right! ;)

Shouldn't we really be trying to get to the bottom of who is behind this DDoS attack? If Wikileaks is said to be a terrorist organisation, isn't this an act of war?

I don't understand the logic - why would attacking a terrorist organization be an act of war?

Is it not?

Reply on news.ycombinator.com