If only mathematicians would construct an algebra of linear transformations. Then the browser vendors could simply implement the laws of this linear algebra once and for all.
I take it that question is roughly: given two matrices A and B, guess which continuous path M(t) of matrices with M(0)=A and M(1)=A the web developer intended or will find most visually pleasing. I don't think linear algebra has any mind reading built in.
By the way, why don't browsers just let you specify the function M(t) to be used for the transition?
The shortest path depends on your metric. There are straightforward choices for translation (uniform L2 norm) and even scaling (log) and rotation (the amount you rotate by), but when all three are combined in one matrix, units come into play, and things like skew and perspective make them more complicated.
This path is not guaranteed to be unique. Uniqueness fails exactly where two or more geodesics cross. The set of places where this happens is called the "cut locus". The example given elsewhere of a 180 degree rotation is a good illustration of a point in a cut locus.
The set of invertible linear transformations is also not connected. It is partitioned into two components: those with positive determinant and those with negative determinant, and there is no continuously differentiable path from one to the other that does not cross through a matrix with det=0 (i.e., that is not invertible). http://dbaron.org/css/test/2010/transition-negative-determin... has some examples of animations between 2D transformations where the sign of the determinant differs.
As for intent... this is the reason for the rule pornel gives in another comment: if all of your transforms use the same functions in the same order, then interpolation is done on the arguments of these functions (this also eliminates the problems with having to choose a metric described above). That gives the designer a reasonably simple way to express intent.
I think that a language for specifying M(t) that is expressive enough to cover all the bases would be very complicated to define, and probably not of much more practical utility than just defining a large number of matrices and doing short transitions between them, which you can do today.
P.S., I work for Mozilla and gave advice to people in the CSS WG when 3D animations were being discussed.
[0] The definition of canonical being, if you locked 10 mathematicians in a room and asked them a question, it's the answer 9 of them would give you.
You can't connect a matrix of positive determinant with one of negative determinant without going through non-invertible matrices. (EDIT: I somehow missed that you made the correction about disconnectedness yourself in a later paragraph.)
Even restricting to matrices of positive determinant (the space of which is connected), I don't think there are always geodesics between any pair of matrices. For example in the 2x2 case, I bet there is no geodesic from I to -I.
No no, the stuff from the mathematicians probably wouldn't work and we'd be left with a dent in our egos.