It is strange one, but try to disable SMT and limit cores to 3 while playing dolphin. Solved all of my issues. Guess linux version have some issues when limiting cores.