I've been trying to piece this field apart for a while now, so here's my take on this:
Each manifold is just a collection of points that can be locally mapped to some coordinate system. Furthermore, distances between points can be characterized by a given metric and paths across these points define tangent spaces of valid directional derivatives.
But that's not as important as the idea that the points composing the manifolds in question represent distinct probability distributions. For example, 2D surface could be made that represents all normal distributions. One dimension is the 1st parameter (mean over all reals) and the second is the 2nd parameter (variance over pos reals).
This let's you think about quantifiably describing the differences between probability distributions of a given parameterization. It's cool and way more complicated than what I've described here. Check it out! Go slow and write out the definitions as you go along.