I have worked for places with amazing culture and benefits. I still left and went somewhere else for a 50% pay increase. At the end of the day I look at the extra amount I am saving and what it does to my retirement account. The place I left to was by no means horrible but was not at the same level in terms of culture/vacation.
Also I did end up returning to that company after 4 years and was able to get an even bigger raise upon my return. As always the best way to increase your pay is by leaving, so sad to be honest.