CPUs are like taxis and GPUs are like buses. You have 64 people sitting in your bus all taking the same path. As soon as two people need to arrive at two different destinations (branching) this whole system breaks down. Eventually the bus must drive to all destinations. Then there is the time wasted stopping to load passengers. To efficiently utilize the bus you must load 64 people at the same. If you were to let people hail the bus like a taxi it would constantly have to stop to just load a single passenger while the other 63 people in the bus have to wait for that single person (random memory access).

A pointer can point to object of any arbitrary sub type in any memory location. Suddenly you have both branching and random memory access. If your program is branching heavily then you may end up with 1 piece of data being computed by a core that can handle 64 pieces of data at once and random memory access kills you because GPUs don't have huge caches like CPUs. The way GPUs deal with high latency is that they simply batch a large amount of work and switch to a different "thread" during memory access. If all your threads are busy loading from memory then you won't see any speedups.

Finally GPUs do not actually have 4096 cores. The RX Vega 64 only has 64 cores and those cores only run at around 1.3GHz. A Ryzen with 8 cores running at 4GHz with out of order execution can trivially outperform a GPU that is only using 1.6% (1/64) of its theoretical performance.