Not necessarily. There's two interpretations to the question: 1. QC would be very efficient and mine efficiently. 2. QC would break SHA and would be able to reverse the hashing function at O(1).
In scenario 1. The difficulty would increase. The mining rate globally always stays the same. And the voting power would be distributed amongst the holders of the new compute, this has happened before with ASICs. Usually there's some graduality to it, and the capital is distributed so that there is never a 51% monopoly. It's especially relevant how big the jump is, if the new computer is stronger than all of the existing miners combined, then they get 100% theoretically (although with malice). In that case there would probably be a fork or as you put it, BTC would collapse. However, if you have that power, holding BTC is probably not that important anyway. The actual compute is worth more.
On scenario 2. Yes BTC would crash, but then again the actual compute power would be more impactful. BTC would crash but so would encryption, and planes and the world.