https://www.newscientist.com/article/2396510-mathematician-warns-us-spies-may-be-weakening-next-gen-encryption/
Quantum computers may soon be able to crack encryption methods in use today, so plans are already under way to replace them with new, secure algorithms. Now it seems the US National Security Agency may be undermining that process
Asymmetric key exchange works by utilising a complex math equation involving massive exponents that is easy to run to get an answer, but very hard to use that answer to get the numbers you started with.
With traditional computers, you essentially need to try every combination of numbers through trial and error to get the starting values.
Quantum computers are almost purpose built for this kind of math and can solve those types of problems exponentially faster than traditional computers.
However, for a symmetric key, there isn't an exchange that can be attacked, both sides already know the key.
There is still a quantum attack against symmetric key crypto like AES, but it just reduces the effective key size by half. If you use long enough keys (256 bits) you're still fine.
Others know a lot more about this then I do. The short answer is that some problems that are hard to solve now can be easily solved with quantum computers. Factorization of large integers is one of those things that is easily solved and public key crypto is based on that sort of problem. Hence the panic. Symmetric key crypto is not.
I think it comes down to just because quantum computers can solve some problems easily it does not mean they can solve all problems easily. Each situation needs to be looked at.