Xeon Phi is somewhat misunderstood, and its early incarnations have not helped that. The most important point to understand is that it is neither a CPU or GPU architecture. Code designed for either of those architectures will be very suboptimal even if it does run adequately.

The Xeon Phi is a modern reincarnation of a barrel processor. It understands x86-64 opcodes but your data structures and algorithms need to be designed quite differently than vanilla CPU code if you want to optimize throughput. No one is doing that though.

In principle, if your code is correctly designed for the architecture, the Xeon Phi should significantly outperform both CPUs and GPUs for a wide range of use cases. The strength of these architectures is that their throughput is relatively insensitive to both latency and lack of trivial parallelism, which are the major bottlenecks to a lot of modern software performance. It is why Intel resurrected this style of computing architecture.

Basically, very few people know how to design software for these architectures even though it is pretty easy (much easier than GPU code). I have experience designing software for exotic architectures like this, I know what they are capable of in terms of throughput, but have never worked with a Xeon Phi. Nonetheless, the silicon specs suggest that someone that actually knows what they are doing should be able to significantly outperform e.g. GPUs for things like machine learning. Right now, it is basically being wasted because most developers treat them like weird CPUs.

I'd love to play with one of the new Xeon Phi processors to characterize its true performance but I am unlikely to see one. But I would not dismiss their performance; it is an extremely efficient kind of architecture for a surprisingly wide range of workloads if used well.