You could put "echo" marks all over your .zshrc file to figure out which parts are taking the longest to execute.