PagerDuty: You Have The Power
blog.pagerduty.com
blog.pagerduty.com
And here's its awesome API in action in Datadog -http://www.datadoghq.com/2012/07/pagerduty/
Before PagerDuty we were using Nagios with a modem dialing out to pagers. Later we went to Nagios with email to SMS gateways. We had "problems" with email to SMS delivery and no way to tell if it was a bad employee or flaky cellco gateway. We also had problems with scheduling on call rotation & overrides for vacation.
The escalation/accountability & reporting let us weed out a bad employee that was ignoring pages. We suspected it but phone call delivery with confirmation in Pagerduty ensured there weren't any plausible excuses.
Most importantly it makes scheduling oncall rotations painless.
Yes, you can do all this in plain old Nagios with enough time or hackery. We reached a point where we couldn't waste time on that kind of hackery (changing alert schedules & escalation every time someone went on vacation, google calendar to nagios config scripts) and needed it to just work.
These APIs are just icing on top. I have two projects that will benefit immediately: 1. Our customer service manager has been asking for a easy way to tell who is on call and we didn't want to give her a pager duty login. Now I can make a simple "who is on call?" dashboard so the call center knows who to call if there's a problem monitoring missed. 2. With the Integration Triggers we can open Jira tickets for incidents. This will streamline workflow.
http://blog.pagerduty.com/2012/08/how-to-provide-24x7-phone-...
If you ever find yourself asking this question, consider the follow-up: "Is my business sought by PagerDuty?" If the answer to your question isn't screamingly obvious the answer to my question is.
n.b. This is OK. In a world with X customers and Y firms there are X x Y possible commercial relationships and the overwhelming majority of them will never happen. That's the nature of the beast.
What's your downtime cost?
What's your implementation (and testing) cost of your home-brew alternative?
How are you going to document your homebrew alternative?
Odds are good that your DevOps hire will be familiar with Pagerduty (or something like it, though probably worse).
PD works, is straightforward, has plugins to other alerting tools (Nagios and New Relic off the top of my head), has multiple contact methods (SMS, MMS, eMail, phone).
Pagerduty is out there. It can't be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until you are dead. Or your site comes back up.
Just a happy customer (despite their core competency being waking me up at 4am -- but given that that of itself is a momentous task, that they're good at it says something).
1) quite reliable (although not as reliable as my production app)
2) totally uncorrelated from failures in my own infrastructure
3) tuneable to eliminate false positives, get good confirmation from users that they've taken issues. An iOS app is the obvious solution here, smarter than IVR + SMS.
Even better:
4) tied into my ticketing system
5) tied into my config management/ops system, so I can let people have simple app/web tools for "roll back! roll back!", etc. At 3am if you get paged, it is nice to have good tools for that kind of thing.
I haven't found anything which is #1-3 without roll-your-own, and #4 and #5 I've never even built internally (but may this time).
If PagerDuty really is #1-3, then it is a bargain at $18/mo.
- a reliable notification system with multiple notification channels (iPhone/Android push, email, SMS, voice, etc)
- integrated with IT management systems (monitoring, ticketing, etc.)
- can provide full context with the alert like performance charts, change history, etc., and not just "something went wrong"
- enables recipients to initiate investigative/corrective actions, communicate with others, etc.
We've build iPhone/Android apps to take advantage of push notifications as well as ability to present rich information and configurable user actions.
I'd love to get your feedback and collaborate to develop it further to meet type of requirements you mention above.
Disclaimer: I'm one of the founders of OpsGenie.
- When users receive notifications, they are often outside the office, and for many enterprises connecting to the corporate network from a mobile device is not practical (even if it's technically possible). A cloud based service ensures that the recipients can access the information easily wherever they may be.
- Cloud based service enables cross organization collaboration without complicated infrastructure changes. OpsGenie enables incident focused collaboration. Only the specified recipients have access to the alert and supporting docs. For example, we use it to receive alerts from customer implementations and can collaborate with operations folks without having to have access to their internal operations systems, etc.
Yes. We use them at Netflix.
The real value that they add is they take care of the edge cases you haven't thought of yet. :)
The text of their notification e-mails is ridiculously bad ("You have 1 triggered Incident", not customizable). The mails don't even contain direct links to the incidents, much less links to ack/resolve them...
There are no notifications about status updates at all, not even an e-mail. People still call each other all the time to synchronize about "Are you on it or should I take it?" and "Is it resolved yet?".
etc.
I'm still baffled how they managed to ruin such a trivial product so thoroughly.
Needless to say we went back to a simple shell-script with TSV schedule (and that amusingly works better in most regards).
Disclamer: I'm one of the founders of OpsGenie