I think that's a fair assessment. If people can grasp creating small independent deliverables for their projects (of any size) then things become a lot easier to estimate. Monitor velocity and pull less into future sprints if you're still getting it wrong.
If there is a piece of functionality that brings unfamiliarity to the delivery team then create a spike ticket to investigate (though use them sparingly).
If you break things down small enough, really understand your velocity and have a transparent relationship with your product owner then there should be no need for any stress from anyone.