You're right that if your server is very slow at generating the response, turbolinks is not going to add much in terms of perceived performance. Same is true if you download multi-megabyte pages. So don't use it for either of those cases!
We use it for pages that have low latency (50-100ms) and a light weight (<100KB) for Basecamp. It makes a big different for apps built like that. This is a project that works with and encourages you to build apps like that.