My wife found letting the baby cry out very difficult so I did have to be quite firm about it...
But my daughter is 8 now and doesn't seem the worse for it ... As far as I can tell.
Edit: only mentioning this in case anyone is in despair and wants to try something different.