Hacker News Reader

I notice a few blips now and then, too - it's visible on my graphs. Sometimes they can be as long as an hour or two. But then, I'm not getting any user reports in these times which makes me just wonder about the monitoring.

I was having some major problems with a server that would flip out under heavy load. Turns out it was the specific kernel I was using. Linode support was quite helpful once I actually contacted them about it.

I also get the blips which are visible on my graphs.

If you have 25+ servers, aren't some of them in other data centers and configured in some sort of high-availability setup? What's your technology stack?

(FYI, I'm asking purely out of curiosity)

We have a backup set of servers at a different data center as well. But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center. A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue.

The only real solution to this is BGP routing which is complicated to setup and VERY expensive (only big guys do it). Here's some excellent answers on SO: http://serverfault.com/questions/153576/global-high-availabi...

But routing traffic via DNS change to different data center isn't the best approach as some ISPs cache DNS and would still send traffic to old data center.

All resolvers cache records for the TTL (or SOA minimum for negative lookups) specified by your authoritative servers.

A solution to this would be to have a round robin DNS to different data centers but that is still plagued if one data center develops a network issue.

If the clients are web browsers they will try the alternative addresses in a record set if the one they pick doesn't respond.

Forgive my ignorance, but isn't it normal to cache the DNS for 24 hours or so by default? I think it's typical (though I've never done it, so I'll apologize again if I'm talking out of my ass here) to drop the TTL down to basically nothing well ahead of the migration so that when you flip the switch, everyone will notice almost immediately, and then you can push the TTL back up to normal on the updated record.

Huh. I have a number of our pre-launch hosting customers on a web server in the Newark center; I get a text message within a few seconds of an outage, and haven't received any notifications in quite a while (unless it was due to my own error).

Maybe it's something on your particular rack or node? Have you tried setting up new Linodes in Newark and migrating to those?

Which data center do you use? I have not experienced any outages with my linode server(s).

How regular and how long? They don't provide 100% uptime guarantee. A few hours a month of downtime is what 99.99% uptime. That's not bad at all.

I personally haven't had issues with the network in the Dallas DC.

There are 30*24 = 720 hours a month. If a "few hours" is 2 hours, then that is 718/720 = 99.72% uptime. This means 0.28% downtime vs 0.01% downtime. IOW, almost 30x worse than a 99.99% uptime promise (if that's even what Linode promises).

Math FTW!

ok.... seems like I can't enter numbers into irb ;p you are right 99.99% would be about an hour a year.

Network outage for us even for couple of seconds can be bad because we are serving about 1300 requests/second at peak which can mean a number of users getting affected.

The Internet itself is inherently unreliable. Your users will traverse multiple hops (many networks) before connecting to your servers at any provider, as will any external monitoring systems you put in place. Brief issues along the paths between them and your servers can and will occur. While the Internet as a whole is designed to route around problems, this doesn't happen instantaneously in most cases, although it can happen very quickly.

This is not to say that internal network issues won't happen at any provider on occasion as well, because they certainly will. No network is perfect, least of all the massive system of interconnected networks known as the Internet, and you need to accept and plan for this. To believe or expect otherwise is simply unrealistic.

We use the Newark data center and monitor our server with Nagios. In the past 90 days I have not received an alert because the connection was down.

My Linode in Newark has not had network issues since last year. Acquaintance has a large deployment there with 100.0% availability. Positive this is not problems with your stack or monitoring?

No, it isn't. In addition to using Pingdom, we are continuously monitoring availability from multiple different locations (these scripts also do DNS switching). Every time we detect an issue, we contact Linode and many times they confirm network issues (sometimes internal, sometimes upstream).

Just two days back they confirmed one internal outage. See this screenshot: http://imgur.com/jYzuD

Almost every time we contact them about an outage, we get this drup response: "Apologize for inconvenience" which is okay, but they should at least upgrade their infra or take steps to avoid it.

Maybe your acquaintance is not monitoring the deployment with higher frequency. We monitor our stack multiple times every minute.

Pingdom is notoriously unreliable and I base this on personal experience and experiences of others. False positives are the norm and it is garbage.

Maybe, but we do monitor using our scripts as well.

I am pretty happy with Linode for the most part and haven't experienced many issues. However when we have had issues, diagnosed the issue and filed a ticket we get always get told it is a known issue and "Apologize for inconvenience".

Why do I have to contact them to be told they are having an issue? Why can't they let me know asap so I can mitigate any problems they are having.

The last issue we had was their backup system was failing and by the time we discovered it, it had been broken for days.

You can check their status page at http://status.linode.com/ or drop by irc.oftc.net in the #linode channel. They also usually tweet as well.

None of which they had done. And their backup system isn't exactly something straight forward to setup monitoring of to notice there was an issue.