I used fish for two years or so, and recently switched back to zsh, mainly because Fish is too slow to load, and this is not something the developers are willing to fix. The problem I had was that I needed to load a “bare” shell, without plugins or anything else, but this is not functionality fish provides, and judging from the developers’ comments something they are not going to.
So I spent a day migrating my fish config to zsh, and am much happier now.