Actions that require database downtime/restarts can happen during the maintenance window. For example, if you change a DB parameter that requires a database restart, you can tell AWS to restart the DB during the next maintenance window. Same goes for DB engine upgrades and the like.

If you're running anything production-worthy, you'll be using one of AWS's multi-AZ/failover solutions for your production RDS instances. In that case, the database will perform a failover in those instances so there isn't downtime.