You certainly can. The thing is though that the hops surrounding the server are generally the fastest hops in the whole path. The server is sitting in a big datacenter somewhere with multiply-redundant, professionally-managed network connectivity.

Doing a direct shot from client to client would be some faster, but not much. Meanwhile there are usually optimizations that are better you could do in the same amount of time.