It's relatively easy to read music theory notation, so that the Gibberish and Goobeltygookus make sense, similar to learning mathematical notation. (I personally believe this is easier than learning the alphabet, but that's a different story)
However, understanding what the abstractions mean is difficult. The difference between memorizing formulas and intuiting patterns. Maybe this is no harder than understanding the abstractions behinds words.
Like most "hard" things, the largest reason for difficulty is one of obtaining persistent practice.
If it were common to sing piano music using theory words rather than a single "lalala", for example, or if our language made use of specific tonal changes (up a second v down a major third), or if everyone learned the theory at the age where we now learn the alphabet, the gibberish goobeltygook would seem "easy".