I have several friends who are doctors or nurses, and have talked with most of them about this. They all parrot the "doctor/nurse-patient continuity" line, but when pressed as to whether fewer handoffs actually outweigh the effects of exhaustion on outcomes, they all seem to take it as a given that it does, without being able to support it.
I suspect that, even if fewer handoffs does take an edge over exhaustion, the handoff process could be improved greatly; from what I'm told, handoff processes vary wildly, often aren't formalized, and even when they are, adherence to process is pretty lax, in part due to the effects of exhaustion.