The exponential captures the idea of slowly perturbing an object by a small amount. Suppose you start with some object, perform some action on it, then add the result back to your original object. On a manifold you can't add objects together, but you can push your object a little bit along a tangent vector (think of the tangent vector just as the slope of a function at the point). If you just recursively do this operation, as long as the amount you push by grows smaller you end up at a well defined point. Effectively you have pushed a point 1 unit along a geodesic (locally shortest path). So you can define the exponential map in much more generality as a Taylor series for more interesting things than just functions on R. For example given a (matrix) Lie group G and its Lie algebra (actually the tangent space at the identity), the exponential map

  exp : g -> G

is defined by its Taylor series

  exp(X) = 1 + X + (1/2) X^2 + ...

(X is some n x n matrix).