I also use the cli heavily for the reasons you've listed. I also use vim as my primary editor so the prompt is only a ctrl+z or :sh away.