1. Requests/second
2. Average Request Time
3. Exceptions
3. Background Queues
4. Mysql/Postgres servers
5. Uptime
6. Memory usage
7. Disk usage
If you have any tips/scripts/tutorials, please share :)
You will have application specific metrics - errors, user counts, message queue sizes. These will help you understand what is happening in the app itself, let you fix problems and ensure that things are running smoothly. For this we use FogBugz to report errors and have various user metrics store in a database.
The next level is the daemons and services used by the application. Apache will report the number of req/s and MySQL can tell you about the number of connections, or the status of replication. These are important for planning growth and ensuring your server capacity is sufficient.
Then there's the servers themselves. This is CPU load, memory usage, disk usage, etc. This ties closely to the services above.
There are a lot of tools for doing these things, but in the end you have to tie everything together yourself. You might start with simple website response monitoring with something like http://www.pingdom.com then move on to having it triggering your own check scripts to alert you when custom stuff goes wrong (e.g. queue sizes).
And of course I'd recommend my company's server monitoring product http://www.serverdensity.com ;) but you could spend some time setting up a bigger, more flexible (and thus more complex) tool such as Nagios to pull all the metrics in.
Edit: Munin profiles services and responsiveness. You'll probably want to complement it with Nagios to monitor service/hardware failures.
And vnstat for realtime and aggregated network traffic. Conky for volatile realtime cpu/ram/etc.
This webpage is a list of servers (front and back end) and their load averages.
At least for all the webapps I run, which serve hundreds of requests per second to tens of thousands of users, load average is a good enough proxy for all the metrics that matter.
When CPU usage on the front end servers is getting high enough that it impacts response times and throughput, that shows up in the increasing load averages. When disk IO on the database servers is getting high enough that it impacts response times and thoughput, that shows up in the increasing load averages.
The web servers each have a webpage on them that has the load average as its only output. I wrote up a little script for the database servers to query the load remotely on the DB servers. The webpage I use as the desktop gadget polls those webpages/scripts and prints out the server names and their current load as its output.
As for uptime monitoring when I'm not around, I used to use AreMySitesUp.com. Nice, simple service that does what it's supposed to. Even with a free plan it seems to check every 15 minutes or more often, and if a site goes down, it checks again much more often until it comes back up. It'll also tell you the exact status code if it's not in the 200-300 range, so you can tell a server is 500'ing out but online, and will notify you of timeouts.
I built my own uptime monitoring service, though, so I don't need that anymore. I run it using VPS accounts at different providers so I don't get stuck with the sites and the service monitoring them going down at the same time. Twilio just added SMS to their API, so adding SMS alerts to any custom monitoring scripts is dead easy now.
I use exception notifier for rails. Simple, but it works. Might switch to newrelic.
From time to time I also use pingdom.com to see that the site is available from different networks around the globe.
2. Again apache logs into ganglia. New Relic for rails. Firebug and yslow when testing.
3. Exception notifier in rails, ErrorDocument in apache that triggers a php script which notifies by email. Log4j in java.
4. Redis, OpenMQ for queueing, Resque for background jobs
5. MySQL only because at the time postgres did not support replication.
6. AlertSite
7. Monit
8. Monit
I am always on the look out for better ways to monitor the servers at work. I never really liked nagios. Monit is extremely easy to use and is very good at what it does. MMonit is a management console that can centralize the monitoring of all servers. Ganglia is great for metrics, flickr can vouch for that one.
We then have secondary monitoring that can allow use to isolate issues within different areas if primary monitoring indicates a problem.
For daemon monitoring and watchdog-ing, I use monit. It emails me if there are problems with the server, and can automatically restart processes if they fail or get out of control.
I wouldn't consider building a production Rails app without it.
I don't know if there are any hosted solutions though.