What monitors the heartbeat server to make sure it is up and running as expected?
What monitors the heartbeat server to make sure it is up and running as expected?
Currently we simply write each heartbeat to our redis database, along with the UTC time it occurred. Some heartbeats are global (string key) and others we have one per server (hash key, server name => last heartbeat). There is an API endpoint set up that some of our systems use to trigger heartbeats.
We then have a public page at a secret url that renders out the full status check. Pingdom is set up to check this page for "warning-status: OK" and if its not there send us an email. It also checks for "final-status: OK" and if that is not there, it triggers PagerDuty which will wake us up if needed.
I would prefer if PagerDuty could check it directly, then they would be our only point of failure.
-- Edit -- I added a screenshot to the post which should make it a bit clearer.