I am interested in physics. I am one of those middle aged men. Not that I aspire to suddenly come up with a theory of everything that people who dedicated their life to it could not think of. I am just curious. I have a solid applied mathematical background with all the usual basics like linear algebra, real and complex calculus, functional analysis, information theory, probability, statistics, ODEs and PDEs, but I quickly realized that I would need to fill multiple gaps in my math before understanding modern physics: quarternions, tensors, Lie groups and algebras, differential geometry etc. so I am slowly working through this.
My problem is: it is pretty hard for me to find a good set of books/textbooks that fill the specific gaps in my math knowledge. I even started to work on DAG of areas and associated textbooks myself but any pointers would be appreciated.