All maths done till ~1600 AD could have been done in 50 years, if you see it from perspective of a civilization which has a stable existence. And forgetting all abstractions, stability of surrounding civilizations is basically a random phenomenon.
And I'm sure maths done in physics departments is beyond understanding of 80% people outside it, let alone any civilization which started 400 years before us.
1. Knowledge that is potentially within our understanding, as humans, even if it takes a few centuries.
2. Knowledge that would require a few millions of years, because we'd need to evolve beyond our current level of intelligence, similar to what chimps, as a species, would need to do if they were to really understand theoretical physics (that is, assuming chimps would in fact evolve to higher intelligence, if there were, say no humans around).
A reasonable interpretation of Elon Musk's claim would be mainly arguing against the existence of a barrier like the type 2 knowledge. But I could be entirely wrong, I haven't read his reasoning behind this claim.