Hmm... Do you have plans of building something at Twitter scale? If you can foresee that, then I guess you are right when you are considering Twitter's case of dropping Ruby.
I am no Ruby-ist but I am sure Ruby is not that bad that you will have troubles scaling. You just have to understand how to scale and you can use either PHP or Ruby to apply the general concepts of scalability.