The performance numbers are not representative of your proposed workload. They are 1) images either from RAM or sequentially on disk that are 2) batched and tested together with vector operations all the way (memory bandwidth also dominates the latency for this reason), and 3) the images are only 256 x 256 pixels (much smaller than a website).
Also, the neural net occupies something like 1.2 gigs of RAM, and something like 3 gigs of GPU memory.