I'd actually say quite the opposite of what you are saying: in a server environment, you often care more about throughput and mean latency (as opposed to 99th %-ile latency), and GC can really do great there. What's nice is that GC allows you to have near zero cost heap allocations and amortizes the cost of cleanup. What tends to be the problem is a lot of languages with GC make it difficult to efficiently work with the systems's buffer cache, so you typically see problems with systems that do disk I/O with large amounts of data.
I've written a LOT of C/C++ & JVM based servers. Particularly for large projects with a lot of independent moving parts and object lifecycles that are not tightly tied to sessions, the JVM based servers are remarkably competitive if they aren't disk IO bound.
There's a reason why a lot of HFT firms use Java...
C/C++ still kicks butt for tightly written servers with fairly fixed functions, and generally for anything that does tons of disk IO (though the Cassandra guys have, as an example, demonstrated one can do quite well if you take the time to get it right), and of course anything with unsigned arithmetic (seriously, when are we going to acknowledge that this was a bad idea?).