You are probably on the right path IMHO. What wad that quote where Einstein said the mathematicians got hold of relativity, and turned it into something he didn't understand any more?
https://www.goodreads.com/quotes/78897-since-the-mathematici...
>Strangely enough no personal contacts resulted between his teacher of mathematics, Hermann Minkowski, and Einstein. When, later on, Minkowski built up the special theory of relativity into his 'world-geometry', Einstein said on one occasion: 'Since the mathematicians have invaded the theory of relativity, I do not understand it myself any more'. But soon thereafter, at the time of the conception of the general theory of relativity, he readily acknowledged the indispensability of the four-dimensional scheme of Minkowski Einstein, philosopher-scientist, Schilpp
And later Einstein expressed admiration for geometry again and again
>This problem remained insoluble to me until 1912, when I suddenly realized that Gauss's theory of surfaces holds the key for unlocking this mystery. I realized that Gauss's surface coordinates had a profound significance. However, I did not know at that time that Riemann had studied the foundations of geometry in an even more profound way. Einstein, Kyoto Lecture, 1922
>I admire the elegance of your method of computation; it must be nice to ride through these fields upon the horse of true mathematics while the like of us have to make our way laboriously on foot Hentschel (1998). The Collected Papers of Albert Einstein, Vol. 8 (English): The Berlin Years: Correspondence, 1914-1918
What he complained about, the Physics textbook approach was the path taken by Einstein who started from the "invariant speed" assumption to derive relativity:
https://sites.pitt.edu/~jdnorton/teaching/HPS_0410/chapters/...
>"introduce another postulate... namely, that light is always propagated in empty space with a definite velocity c which is independent of the state of motion of the emitting body."
There's also nothing really "wrong" with Einstein's approach either, it's motivated by physical constraint from Maxwell's electromagnetism.
Mathematician's approach is motivated by imposing logical constraint of an abstract mathematical structure (group of transformations from one coordinate system to another)
The two turn out to be connected (physical constraint selects a math model from alternatives). Nothing really strange.