John Carmack seems to think otherwise.
> "That means that in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS"
https://twitter.com/id_aa_carmack/status/1503844580474687493...
The OP's comment is that while Python's native interpreter is slow, Python code as a whole can still be fast since people will delegate the hot spots to non-Python libraries optimized for speed.
To refute that, you linked to John Carmack quoting Horace He saying that Python's native interpreter does CPU-based math slower than a GPU. But...that's why people writing Python use libraries to delegate math-intensive work to a GPU, which is the OP's point.
The code that is written in a language other than Python is often not written by the developer writing the Python code.
As an example, I've been fiddling with inferring automatic data extraction rules for data in scraped webpages.
The meat of the algorithm and its heuristics is in Python. The heavy lifting of parsing HTML and evaluating CSS selectors is done by a C parser that I did not write. For me, prototyping the heuristics in C would have been quite a bit slower.