Occasionally there were conflicts or problems caused by plugins, and then I just troubleshot those and uninstalled or disabled plugins until I found the culprit. No big deal.
Whenever I use another computer that just has vanilla vim installed, without my own .vimrc on hand, I just touch ~/.vimrc and away I go. I just try not to spend too much time on such systems before loading up my own .vimrc. Again, not a big deal.
I really don't get people who deliberately cripple themselves by using fewer plugins or features than were maximally useful.
On the other hand, I was never a big fan of giant plugins that make a lot of choices for you. I like to make my own choices as much as possible, and know why and how these choices are made. So the plugins I install tend to be very specific plugins that are good at one particular thing that I feel vanilla vim is lacking.