Puma is great if you have a Ruby implementation without a GIL (e.g. JRuby, which it was built for), but it's not so good on MRI, because you have to use clustered mode, which runs multiple processes. What they don't tell you is that when running clustered mode, the requests are randomly distributed to each process, and then the requests queued at that level. Each process can run multiple threads each processing a request simultaneously, but because of the GIL anything that's computationally heavy will cause the other requests in that process to run slowly.
That means that it's possible for one process to become backed up with requests, while the other processes on the dyno are sitting idle. And it's actually not just possible, but likely when under reasonable load.
So there is actually no deployment configuration of Puma which makes sense under MRI, at least without a reverse proxy on the server handling queuing:
- Single process (no clusters), 1 thread: Bad because you use only one core, but consistent performance assuming you're not on a multitenant 1X or 2X dyno.
- Clustered, 1 thread per process: Bad because requests are queued at the process level, so a slow request will cause any others sent to that process to have to wait. If you have a request which takes 25s to execute, the one sitting behind it in the queue is highly likely to timeout, and this can cascade to the next request, etc.
- Clustered, multiple threads per process: Better, but still bad, because while multiple requests will be running at the same time in the process, they are competing for the GIL and can slow each other down so much you'll still get timeouts.
The best solution we found to all this was to run a buildpack with a nginx reverse-proxy to a Unicorn server. It's nice because each request is run in it's own process, and the requests are queued at the dyno-level by nginx until any of the processes is free. Combine that with Performance dynos, so you can run a small number of dynos with lots of workers each, where each dyno has consistent performance, and the queuing issues mostly go away.
As you may be able to tell, I've spent far more time debugging this behavior than I would have liked to!