Well, you likely have a dedicated (or pretty strong) server.
The Heroku dynos I've tested with some years ago performed quite horribly. Only a proper Xeon server was able to achieve sub-100ms responses.
But hey, if Ruby improved in the meantime, cool.