Your view of Sanskrit is probably skewed by a few things. First of all, nearly the entire corpus of Sanskrit literature was written after (and in many cases long, long after) it was used as a day-to-day language. It was probably already losing ground at the time of Panini, roughly 500 BCE.
Second, what Sanskrit literature we have is highly artistic and edited literary text, replete with hard-to-get cultural references, not ordinary spoken language.
Finally, it's probably just plain different from whatever language you're used to, not intrinsically harder or more complicated.