This is great advice. Another related issue I see is when you engineer a new feature that can't always be 100% accurate because the source data is spotty but you intuitively think the new festure should help the classifier anyway when it is present. And if the new feature's feature importance in the trained model turns out really high, you think you've done something great. But in the end you made model that simply detects the presence of your new feature which you knew wasn't 100% accurate anyway because the source data it is derived from is spotty. So you've accomplished precisely nothing.