Your use case is exactly what we built Phusion Passenger for. It could be that applying some configuration could improve startup performance for you, but general runtime performance of Passenger should be on par with a Unicorn solution.
Note that to use Unicorn you would have to configure an nginx instance that proxies for Unicorn. This is where Passenger shines, we already integrate with nginx so there's no extra configuration or management overhead. Passenger is resilient so it will survive any crashes of your application and automatically respawn any failed processes.
edit: The configuration to make your Passenger perfect for dedicated single app servers is:
passenger_min_instances your_num_of_cpus;
passenger_max_instances your_num_of_cpus;
If you want a more complicated setup, for example with varnish and haproxy between nginx and your application servers, thin is rock solid. It runs Rails 2.x-4.x apps without any issues. It's also well suited for handling websockets traffic.
If the workload of your apps can take advantage of threads (e.g. lots of external API calls, or lots of time spent in the DB), a server like Puma will show an advantage. Also, if memory is limited, you can handle more concurrent requests.
Passenger works just fine with single-tenant dedicated instances. Just configure passenger_min_instances to the same number of max_instances or max_pool_size, and it'll behave exactly like Unicorn's default settings.