For any native speaker of Indo-European languages it is much easier to learn another Indo-European language than to learn a language from a different family, mainly because the Indo-European languages share a lot of their vocabulary, even if that is not always apparent to less experienced speakers, because the pronunciation of the words can differ very much.
Learning unfamiliar words is a problem equally difficult for any language that belongs to another language family, so the difference in difficulties can appear mostly in grammar or phonetics.
The Indo-European languages are among the most difficult for speakers of other language families, because the so-called fusional inflection characteristic for the IE languages, which is very seldom encountered in other language families of the world, is an extremely irregular means of expressing the grammatical relationships between words.
So, except for a few secondarily simplified languages, like English, it is much more difficult to learn an IE language grammar than it is to learn a more regular grammar, such as that of languages like Hungarian or Turkish, or even that of Japanese.
In most non-IE languages that are considered difficult, the difficulties are not in grammar, but in phonetics, when those languages use sounds seldom encountered in other languages, e.g. tones, clicks, pharyngeals etc.