Solutions to linear differential equations are exponentials, and this may be what the author meant. In a state space model, we can write Newton's law under gravity if we let y(t) = [x(t), x'(t)]^T, let A = [ 0, 1; 0 0], let b = [0, -g]^T, and set y'(t) = A y(t) + b.

Indeed, the solution is y(t) = exp(A t) (y0 + int_0^t exp(-As) b ds).

In Einstein's theory, we have a generalization which says a stone's path through space-time follows a geodesic of the metric. In differential geometry, the exponential function, by definition, sends tangent vectors to geodesic curves.

So the author is right, under this interpretation.