“We have been experiencing a catastrophic DDoS attack”
status.linode.com
status.linode.com
I would use Linode if I needed to lease computational power, because it is still a great value vs AWS, but I could not run a high availability service there. It would feel like professional malpractice at this point.
Found the culprit.
Site B, insure any writes use even numbers for unique field.
This avoids conflict.
If I'm updating existing records, and the MySQL master at Site A gets updated, then goes down before Site B is updated.. I've got an inconsistent setup.
Been thinking about Master-Master MySQL replication recently as we have a system that's duplicated and taken offline each summer (to run a summer camp), and looking for a way to sync changes in it back to the main 'live' MySQL database.
That's master/slave, but to get master/master, all you need to do is start a slave on the original master and point it at the current master position on the original slave (which should be static since it isn't yet accepting any queries directly). These posts may be helpful:
https://www.digitalocean.com/community/tutorials/how-to-set-... http://plusbryan.com/mysql-replication-without-downtime
Once it's running, as long as you're not running autoincrement queries or other things that can conflict on both servers at the same time, without taking appropriate precautions, it should chug away without any intervention.
If something does go wrong, you can often figure it out by looking at the slave status, fixing the inconsistency manually, skipping the bad query, and then starting the slave. If not though, you can always just re-synchronize from scratch. Or even better, run your databases off of an LVM volume, then take regular snapshots. (IE snapshot, make a tarball of the snapshot of the mysql directory, then remove the snapshot.) That will give you a consistent backup, even with the server running. On an SSD, the temporary added latency probably won't be noticed, especially if done off-peak. Then if anything goes wrong, you can restore from the snapshot, and it should catch back up to the master from the snapshot's position automatically (as long as your expire_logs_days setting in my.cnf is longer than the duration since the snapshot was taken).
Hmm, may have to give it a try! Thanks.
It can be installed as a plugin [1] and there are patched 9.4 packages available [2].
I set up a POC with databases in UK and US (east coast) last year. It was fairly straight forward and seemed to work well.
1: https://2ndquadrant.com/en/resources/bdr/ 2: http://bdr-project.org/docs/stable/index.html
I switch between either master with just a DNS update that propagates within 5 minutes.
If you use mysql's read_only flag and application users don't have SUPER permissions, you can easily prevent writes to the wrong server; set read_only = 1 in my.cnf and manually set it to 0 on exactly one of the masters. Use the read_only flag to drive automation for which server to send writes to.
Manual failover is set old server read_only, kill existing connections (read_only flag is cached), wait for replication to catch up, set new server read_only = 0. You can make a script to do this with one button, but I wouldn't recommend making it autonomous: flapping between servers is disruptive and could lead to data inconsistency if you switch when replication is behind; data inconsistency is usually way worse than write downtime until someone logs in to flip the switch.
Try to have half your slaves off each master, so if a master is down, you still have 50% capacity. (I've seen some patches from google a while ago to keep binary logs in sync between masters, and make switching masters easy: If that's available, you may be able to have slaves just follow the current active master)
If you have budget for it, an extra slave off each master can be helpful: You can cron them to shutdown MySQL, tar up the directory, and restart. If you untar that on a new slave, it'll continue replicating from that point in time. If you rotate out the backups, you also have some ability to restore data from the past, if there is a bad update.
How have you found Vultr?
They seem to be the next biggest name in VPSes after Linode and DigitalOcean, with broadly similar price and quality.
I am with all of you guys that are affected by this. I am looking forward to them to resolving this soon.
My understanding is that Atlanta in particular has some poor upstreams and is making our job pretty difficult there. Notice that it's almost always Atlanta that's getting hit. I would suggest just using other datacenters or making sure your high availability model includes several DCs (which I would suggest at any hosting provider, really).
XSAs are released alarmingly frequently. I've always wondered if KVM is really more secure or if it just gets less scrutiny (certainly there's a class of qemu vulnerabilities that impact both hypervisors).
It would appear? you'd need to go back for quite a few months of being unpatched to find a genuine issue. Unless something about Amazon's mitigations don't apply universally.
That said, I don't disagree that the attackers might be trying to distract the team while they exploit that... though I don't see what it gets them compared to quietly exploiting the XEN issues before they were common knowledge on 9/8.
The amount of distributed traffic happening right now against linode would probably only represent a 5% increase in traffic to a popular Google product. At least you know they have the expertise. Nothing against the very smart and talented linode engineers, but the two companies are on very different levels of traffic engineering.
But if your service is the direct target of the attack (as opposed to the whole provider) and your servers are getting hammered...
Even if you architecture allows quick horizontal scaling, you still face a tough decision.
The attack could go on for days, and the hosting costs can go really high really fast. Which can be catastrophic for a small company.
Google/AWS probably have 100 times the capacity (and redundancy and architecture reliability and failover and awesomeness) of linode. That means that, first, they can't be put down easily, second, a DDoS is limited to a small subset of the infrastructure and doesn't bleed to every customers and services.
As for traditional hosting companies (OVH and the likes) When you're being DDoSed, they'll null-route your IP space. (i.e. they advertise your IPs as dont-exist-on-the-internet-anymore). The traffic is dropped while in transit on the internet because it can't go anywhere. It doesn't reach the hosting company anymore.
Note: being null-routed means your site and all your services are off the internet and thus effectively dead.
As for CloudFlare. They have many locations all around the world and they can absorb a lot of traffic, to the point they themselves cannot be DDoS. They have active monitoring and mitigation against common attacks and known malicious sources, which may prevent the attack without even you knowing about it.
When you're under attack, you can block subnet/AS/countries in cloudflare settings, or request a challenge/capcha from every visitors. Cloudflare will reject visitors (with or without challenging them) at their edge location before any traffic can get to you. It is very effective from my experience.
Generally speaking. The only way to stop a DDoS is to do it before it reaches your datacenters so you need help from your ISP/provider/CDN.
Edit: The attack that put down linode last christmas was against linode itself and not a specific customer. Part of the mitigation included linode moving its critical services behind cloudflare :D
OVH hasn't been doing this for a while, they got some beefy ddos protection setup for this exact reason - it was way too easy to take down someone for hours
Hetzner (another big european hosting provider) followed recently: https://news.ycombinator.com/item?id=12403783
Online.net also has included protection (+ paid upgrades)
At least here in europe the big hosting providers are all switching to providing included protection for all their customers, at least for traffic intensive attacks which hurt everyone
The smaller hosting companies may still very well go out of the game if the problem worsens, even if most DoS venues do end up being mitigated. I don't know how I would respond to that as of this moment, but hopefully it doesn't have to come to that. It's already tough to find a decent hosting company in my experience.
https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...
But could it simply be severity of the attacks? I keep seeing comments about a 2 week ddos attack last christmas - that's something that i would be shocked to see Google/AWS succumb to. Not that Google/AWS attacks don't happen, i just can't imagine them being down for ~2weeks
(I imagine it was just one datacenter from Linode, not the entire service, fwiw)
One thing that took them so long was that their upstream ISPs at some of the datacenters were themselves unable to handle the DDOS, so they had to switch ISPs, which took a while.
I don't see Google/AWS as easy to attack; but I'm not sure why similar tier players like DigitalOcean aren't being hit -- or maybe they're just less transparent about things, or are actually a smaller target (didn't think they were?).
edit: here's a postmortem from linode of the christmas attack - https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...
Linode has a bunch of great features, but after seeing them get hacked a half dozen times over really silly things, more DDOS's than you would be happy with, and frankly I have had interactions with their management (just online) and was sorry to have had said interactions.
You can also read a bunch of implications from former employees about their management, but you can feel free to discount that given how many times ex-employees are a bit pissed.
- Former Linode Customer
which was caused by our own maintenance several weeks apart (the root cause description is really quite good).
I think the distinction people make implicitly is a 25 minute outage versus 8 hours. DDoS attacks suck, but they're just standard these days. As a customer though, any source of (network) outage usually has the same outcome: "my site is unreachable (and I don't care why)".
The reason we (and AWS and others) offer multiple datacenters/zones within a <1ms boundary (a compute "region") is so you can build a highly available app that can fail over with minimal degredation. For customers that were using App Engine Flexible environment with the regional spreading turned on, only some of their instances were affected, but their apps shouldn't have skipped a beat.
Linode is good at what they do, but any customer in Atlanta just had to wait this entire event out.
Disclosure: I work on Compute Engine.
[1] http://highscalability.com/blog/2013/8/26/reddit-lessons-lea... [2] https://www.4chan.org/news?all#106
Edit: I guess an attack that causes your infrastructure to auto-scale could get expensive REAL quick...
That's pretty harsh.
I would still recommend them as we have had great luck with their Dallas DC. It seems to be the most resilient of them all in terms of random network outages and also DDOS attacks.
However the one in December did take them offline for a few hours but after which they apparently implemented more DDOS protection. Keep in mind this could happen to anyone (Digital Ocean, Vultr, etc) and mostly their mitigation techniques seem to be to kill your VM until the attack is over.
That is a bit of an understatement. Our servers were down every couple of hours for a week, and the attack bounced around to pretty much all of their datacenters.
Thankfully it was over Christmas time so our clients were mostly offline.
Is there a list somewhere that shows what the capacity of each of their datacenters is?
Atlanta got trashed the worst back in December but they beefed it up a lot. But how much? Is Dallas still bigger?
Im considering migrating everything to dallas.
Dallas has been superb though, definitely go with that if you can.
I don't understand this line of reasoning. It's not like DDoS attacks are some kind of 0-day failure mode that nobody has seen before.
Would you also say "it is not their fault" if their uplink provider had a fiber cut and they didn't have redundant uplinks? I'm guessing not: it's well understood that has a service provider you need to plan for this kind of unavailability and pay more money for redundant links. So it seems really weird to have this double standard for a different kind of availability failure mode.
Just like network availability or datacenter power availability, you need to invest technical and financial resources into DDoS defenses if you want to be resilient to incidents. If you don't do that as a hosting provider, I definitely won't feel sad for you.
Edit: Not sure what alternate reality the downvoter lives in, but vast majority of the attacks these days are just "dumb" packet floods or even easier to filter reflection attacks. (Linode clarified this to be a mix of dns and ntp traffic on IRC)
But hey, go on and find me a layer 7 attack that'll take down entire datacenters :)
I'd imagine it's easier to filter out bogus L3 traffic when the vast majority of your target IP space comes with explicit configuration as to what sort of L7 application traffic is acceptable.
It's actually not that easy to Filter "bogus" traffic. In the hosting world, especially cloud, you have thousands of customers doing whatever they want. Who knows what is bogus or not. And even if you can filter it at your edge routers your transit links are still going to be getting slammed. The filtering needs to be done upstream in the ISP network. This is usually a manual process as no one supports BGP FlowSpec at the moment.
RTBH is the best way to defend if you don't have the bandwidth to absorbe.
Put an ELB in front of the services, the ELB only listening to port 80/443, roll out the ELB publicly, roll out new instances only accessible privately, kill old instances being DDoSed => Problem solved => Repeat for all other services, they shouldn't be publicly accessible in the first place.
Ain't saying it's easy but there are some options to help mitigate the attack.
source: been on the receiving end of a several hundred gigabit L7/HTTP DDoS from the aforementioned providers.
That's for volumetric attacks (udp reflection), tls handshaking can eat all the CPU way before we run out of network :(
Vultr had problems a few weeks ago, but I don't think it was DDOS-related.
Somehow things have been quiet on the DigitalOcean's end.
This leads me to wonder: How much do other providers leave customers in the dark?
Also, we're on Heroku and they advertise Ddos mitigation as a feature, but "mitigation" sounds non-commital and I'm curious how they'd fare against a similar attack?
Mitigation sounds non-commital because it is. I don't personally have any experience with mitigating a DDOS attack on Heroku, so I'm not qualified to talk about their prepardness, but DDOS mitigation varies quite a bit [2][3], running the entire gamut of "block a single IP address" to "there's hundreds of IP's rotating and attacking". So the answer to that would be: it depends on who you've pissed off or who's feeling particularly nasty that day.
[1]: https://www.cisco.com/c/en/us/products/collateral/security/t...
[2]: http://www.continuitycentral.com/index.php/news/technology/6...
[3]: https://www.radware.com/newsevents/mediacoverage/changing-ch...
If you look at the history of Linode discussions under HN, most of them were related to DDoS attacks and service downtimes. https://news.ycombinator.com/from?site=linode.com
I hope they will recover soon and make the service stable.
But now I think I need to setup fail-over with another VPS provider. What's a recommended alternative? Is Digital Ocean the next best choice after Linode?
From what I've heard, the FBI will collect a whole bunch of information and then sit on their hands because of the above reason.
Tor exposes a SOCKS interface, you can't control TCP/IP with the level of detail needed to perform these attacks.
FBI can probably monitor both sides of those nodes (tap the data centre?) if they're in USA? So can't they monitor all the nodes clients, then do something like block returning traffic and look for timing or other meta data of the re-request from the command server?
Anyone tell me how reasonable that is?
An oxymoron ?
The means of really combating a ddos is costly and extensive, this is why most use a service like prolexic or silverline, and typically with some massive infrastructures that comes with it. Anything less than n-by 40gb disposable internet pipes, preferably regional as you are, you can/will be smited at will.
Going cheap with one of the non-big 3 public cloud providers will cost you.