I get why it might not matter depending on the work you do, but if non-trivial work is required just to keep your code building due to fast non-backward compatible changes in libraries you use, that might introduce hesitation.
Isn't that the point of semantic versions though? You can lock it if you need to.