There's also some more common features in most IDEs like being able to jump between symbols that saves a lot of time day-to-day.
If I'm interviewing you and you say you don't like using an IDE or a debugger, that speaks to your work experience, your productivity, your self awareness about your productivity, and really puts an upper limit on the difficulty of the problems you've had to solve.