Your intuition is basically right: perfect octaves/fifths/fourths are called "perfect" because they have the most acoustically pure: 2:1, 3:2, and 4:3, respectively. They're also the oldest consonances, described supposedly by Pythagoras but extant in the Euclidean "Sectio canonis" (~3rd C. BCE).

The other intervals weren't considered consonant until much later (the middle ages), and didn't get their "major/minor" names until a bit later, in the Renaissance. Before this, they were known by their Latin names (M3 = ditone, m3 = semiditone, and so on). The m2 was found in (at least) two forms until the advent of equal temperament, usually known as the major and minor semitones.

The "major" intervals are those found in the major scale (M2, M3, M6, M7). The minor intervals aren't named in relation to the minor scale, but rather in relation to their major equivalents (a minor second is lower than a major second). "Minor" comes from "molle," which is "soft": the major third is "softened" to become the minor third, and so on.