That's quite possible. There are many problems which are said to be NP-hard, but are only NP-hard for the
worst case. Linear programming, for example. Some cryptographic algorithms are much easier for some keys than others. There are known to be weak keys for some algorithms. Weak keys may be more common than generally assumed.
There's still no problem useful for public key encryption with a provable lower bound on solution work effort. Public-key encryption started with the knapsack problem, which turned out to have an efficient solution. Factoring is assumed to be hard because much mathematical effort has gone into studying factoring with only limited published success. It's not like there's a provable lower bound for it.
The only thing we really know works is a one-time pad, and even that can be screwed up by operational errors. A "one-time pad" used twice is easy to break.
It's worth reconsidering one-time pads; in the era of the 64GB flash drive for $30, you could have enough one time pad for a thousand hours of voice calls on one USB stick.