I've tried all kinds of monitoring solutions, and my favorite is still munin[1]. Out of the box it gives you tons of system-level metrics. It comes with contrib plugins for service-level metrics about Postgres, Nginx, and dozens of other popular tools. Then you can write your own plugins for higher-level things. A plugin only has to print a few lines of text, and you can write it in whatever language you like. I have a plugin for Phusion Passenger[2] and another for blocking Postgres queries[3]. You could even collect more application-level metrics if you wanted.

More things I love about munin:

- It is fast. It's just pre-generated static html files, so you never have to wait for a page to load.

- Simple to configure: the only tricky part is a config file with a list of all your nodes, but that is easy to generate with Chef/Puppet/Ansible/Fog/whatever.

- Information density: Tufte would love it. No commercial tool comes close here I think in terms of getting raw info onto the screen at once. This is so valuable for seeing trends or anomalies. When something is going haywire, munin often helps me find the problem in minutes.

- Lots of graphs all at once: I also like that I don't have to navigate or fiddle with zoom levels to see what I want. Navigation is almost always one click and then Ctrl-F or using the mouse wheel.

- Plenty of history: you can look back at the last year of data to see trends.

- Easy to install: just apt-get. Adding a plugin is just a symlink.

- Your data never leaves your own machines.

I think CloudWatch is fine for AWS-specific metrics, like your EBS burstable I/O budget, but for everything else I'd much rather use munin.

[1] http://munin-monitoring.org/

[2] https://github.com/pjungwir/munin_passenger

[3] https://github.com/pjungwir/munin_postgres_extras