NP hardness is defined in terms of worst-case hardness, not average case hardness. As long as there exists even one instance that makes your algorithm take non-polynomial amount of time to terminate, P won't be equal to NP.
For example in SAT, the distance of the instance to the critical clause/variable threshold. In subset sum, the density of the instance.
It's not the problem class that matters.