Faster settlement means less time to discover and correct mistakes.
At a high level, nobody really likes trades being broken, but it's kind of a necessary "evil" that helps stabilize the entire system.
Let's say settlement happens in one minute, one morning a pension fund fat-fingers a price and loses 10 million USD to some student in her dorm room. That afternoon, the student then donates 1 million USD to an orphanage, pays off her grandparents' mortgage, buys 1 million USD worth of municipal bonds to finance a new baseball stadium, and buys several million USD in long-dated ETF call options. Some market-maker delta-hedges those options by buying the ETF. In response to the price move generated in the ETF, a large institution buys a basket of stocks and participates in the creation process with the ETF's issuer.
Now, what happens when the fat-finger is discovered and disputed? Too many things have changed for the SEC to break all of the trades and other transactions after the settlement. The student no longer has enough money to make the pension fund whole (even if everyone involved could agree that the most fair thing to do would be to limit the student to 100k profit and transfer the rest back to the pension), so I guess they're stuck with the pensioners having to live with that 10 million USD loss. "Oops, next time, make sure your pension hires a more careful trader, I hope you've learned your lesson" isn't very helpful advice to a retiree who's now out on the street.