Because the adjustment window is not real-time-based, but block-based: every 2016 blocks, an adjustment is done - with the obvious caveat being that the duration it takes to mine 2016 blocks is based on the global hash rate. Also, the downward adjustment is limited to (IIRC) halving the difficulty at max.

So, given the number of miners drops too far too fast, Bitcoin may end up in a situation in which the global mining power is too low to even reach the end of the adjustment window in a realistic timeframe, which in turn triggers even more miners to shut off their equipment and sell their coins, which lowers the price even more and lengthens the timeframe, which in turn...and so on. This situation is known in Bitcoin circles as the "chain death spiral", and it has been a purely theoretical thing - until now, at least.