• The GPU in the M2 Ultra has 76 GPU cores and corresponds to 2x M2 Max, which has 13.6 tflops; so 27.2 tflops [1,2]
• RTX 4090 has 82.58 tflops [3] (overclocked can reach 100 tflops [4])
While more powerful NVidia cards are not "multiple orders of magnitude more". Rather it seems a 4090 is around 3-4 faster than an M2 Ultra GPU.
Keep in mind that the Apple Silicon chips also have low-precission Neural Engine circuits for inference of neural nets. For the M2 Ultra they claim 31.6 tops [1].
[1] https://www.apple.com/newsroom/2023/06/apple-introduces-m2-u...
[2] https://www.notebookcheck.net/Apple-unveils-M2-Pro-and-M2-Ma...
[3] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889
[4] https://videocardz.com/newz/overclocked-nvidia-rtx-4090-gpu-...