What have other people had success with?
I think Grafana will fill the basic GUI alerting needs, though. When you need more than a simple flat treshold you usually want to get out of the GUI and ask the ops team for help anyway.
And using a single paid tool that does the job better AND doesn't kill me in maintenance work.
See https://www.datadoghq.com/ as leader or https://signalfx.com/ as the second comer, or http://www.bmcsoftware.uk/it-solutions/truesight.html if you're enterprisey.
$15/month/host gets expensive fast. Datadog doesn't start providing discounts till you are at 1000+ hosts.
$15 * 500 hosts = $7500 per month.
If you think it's expensive, I can only advise you to check how much the hardware will costs on EC2 to run the free tools, plus how much work it will take to get the 8 different and independent OSS tools to work not only alone but integrate together, plus how much additional work and maintenance to keep it working without hiccups (war story: there is nothing worse than a monitoring tool that is less reliable than the thing it monitors).
Oh that's ridiculous pricing. Any team running a server installation at any sort of scale would scoff at the pricing, and that's even before you take into account the implications of sending so much of your metrics data to a third party to be held hostage there if you decide to leave the service.