A single floating-point arithmetic operation in mpmath at low precision involves something like 100 "Python cycles" (bytecode ops), each of which takes perhaps 100 machine cycles.
That makes it, roughly:
100 times slower than machine arithmetic in Python.
10000 times slower than machine arithmetic in C (or NumPy if you can vectorize your code fully).
100 times slower than arbitrary precision floating-point arithmetic implemented in C (~100 machine cycles).
These are obviously just order of magnitude estimates.
However, low precision is the worst case, relatively speaking. mpmath uses Python longs internally, and it can also use GMPY when available. At sufficiently high precision, the time is dominated the Python/GMP kernel for multiplying integers and performance is close to other bignum implementations.
Also, for computing transcendental functions, mpmath uses fixed-point arithmetic internally, which reduces overhead a lot.
The biggest problem with mpmath is that it doesn't implement algorithms that scale optimally for all operations, and a lot of the error analysis is completely nonrigorous.
Since 2012, I have been developing a C library (https://github.com/fredrik-johansson/arb/) that solves many of the shortcomings of mpmath. It is obviously much faster at low precision (the factor 100 mentioned above), it generally uses much better algorithms, and it tracks error bounds automatically using interval arithmetic.
1) As soon as you wrap it in a Python class, much of that speed advantage goes away.
2) Double-double doesn't solve the problem with the limited exponent range of doubles.
3) When the goal is to allow setting the precision to any number of bits you like, having a separate implementation for X bits complicates things a lot, especially when the X-bit arithmetic doesn't quite behave like Y-bit arithmetic for X != Y (double-double behaves somewhat differently from 108-bit floating-point).
IMO, double-double would be most interesting as a NumPy extension.
mpmath internally uses Python's builtin long integers by default, but automatically switches to GMP/MPIR for much faster high-precision arithmetic if gmpy is installed or if mpmath is imported from within Sage.
With gmpy you rely on C libraries. Faster but those are still arbitrary precision computionations.Edit: I just tested multiplying two 100 by 100 matrices, and got a 5x speedup with PyPy compared to CPython. Which is nice but not amazing.