That's just because we know very little about them. It very well could be the case that factoring semiprimes might be ridiculously easy, but we are just too dumb right now to figure out how to do it quickly (even if P != NP -- we haven't even shown that factorization is NP complete).