I believe the slowdown is due to the clink prompt running external programs in order to find git status etc. I run clink without ConEmu (cmdr is a pack of both with some config) and recently redid the prompt to make it less functional but MUCH faster.

I basically changed the config.lua file to use clink.get_cwd() instead of pwd(), and to

            local HEADfile = io.open(git_dir.."/HEAD")
            if not HEADfile then return false end
            local git_branch = HEADfile:read():match('ref: refs/heads/(.*)')
            HEADfile:close()
            return colors.git_dirty .. " (" .. git_branch .. ")"

to find the current git branch if any. Less functional and more potential for errors (detached heads or such) but for my case it's very worth it.