The computations used to do symmetric and asymmetric encryptions are completely different. Asymmetric encryption is just about factoring out big numbers into primes, if we simplify things a bit. And modern computers aren't very good at it, but Quantum Computer happens to be, and can break it. Read a bit about Shor's Algorithm[0].
On the other hand, symmetric encryption can be seen as a super convoluted and costly shift cipher. And it seems that Quantum Computing does not help much with dumb and costly mathematics like this.