[For instance, we might assume that a basketball player's score in a game is a linear function of the number of shots they make. The possible states of the world are the multiplicative factors (points scored)/(shots made). This space is simple: it's just a line, probably even just a ray since negative points are impossible.]
One major trick is in the "seeking". To do so we often assume that the space is parameterizable like that it has latitude and longitude and then we scan over all possible choices of parameters looking for the parameters of the optimal point. Depending on the kind of model of the world you're working with, these parameterizations change.
[In the running example, the space is a line and the actual assignment of positive numbers along that line is a suitable parameterization. Higher dimensional models or curved models make parameterization tougher.]
Information geometry uses the tools of differential geometry, the same ones used to characterize general relativity in physics, to characterize this "state of the world" space more completely. It provides new tools for parameterization and understanding when older parameterization tools failed.
[In the running example we don't much need differential geometry to understand the geometry of our ray. In high dimensions, models with interactions, curvature, with discrete and continuous parts, intuition breaks down.]
It also provides a rich geometric vocabulary useful for visualizing the "state of the world" space which can be instrumental in understanding statistics, building new models, evaluating how models compare with one another.