The quality score could go down which can lower rankings with too many broken links.

Google has at least 50-100 variables it checks and assigns different weights to.