> What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card?
Thread divergence.
CUDA-cores are linked and take if-statements and for-loops together. This means that if one CUDA core takes a for-loop 1-million times, 31-other CUDA cores will take the 1-million loop with them. (32-cores per NVidia SM). EDIT: CUDA keeps things semantically correct per thread by "throwing away work" (execution mask is disabled, so those 31-other threads effectively execute NOPs), but the cores are wasted.
Worst case scenario, your CUDA cores accomplish 1/32th the work they could do, as they spin idle waiting for the last 1 thread to complete a for loop or its unique combination of if-statements.
Matrix multiplication doesn't have any thread divergence, because all threads loop the same amount of times on all workitems. This means that Tensors / Deep Learning simply ignores the problem, because they're written as matrix multiplication problems.
Avoiding thread-divergence, or mitigating its effects, is possible, but requires advanced programming skill that few have.
--------
Examples:
Chess -- Traditional multithreaded chess algorithms have every thread check out a different branch of the chess search tree. But because each branch has different positions and moves, its very difficult to code it in such a way that all threads are doing useful work.
Web Servers -- If every thread is handling a different request, then every thread probably traverses a different set of if-statements and loops. This is a thread-divergence nightmare.
Databases -- SQL Databases traditionally have each thread running a different, independent SQL query. Like web-servers, its difficult to imagine parallelism from a high-level.
-------
However, elements of Chess, Web-servers, and Databases can probably be efficiently parallelized onto GPUs.
Chess -- This github repo demonstrates that enumerating positions can be done GPU-parallel: https://github.com/ankan-ban/perft_gpu
Web-servers -- Text parsing, Regex, and many other problems common to Web-servers have been parallelized on GPUs. The main question for me, is whether or not the PCIe traversal would be worthwhile (lower bandwidth than RAM, high latency). Web-servers tend to be I/O heavy and less compute heavy... probably not dense enough to benefit from GPU compute.
Databases -- Merge-join and Hash-join probably can be parallelized to GPUs.
Figuring out the optimal CPU + GPU team is going to be a research problem over the next 10, maybe 20 years.