There was at least one mishap during Apollo capsule reentry where astronauts were exposed to fumes from hypergolic fuel, after forgetting to purge the thrusters at the proper time: https://en.wikipedia.org/wiki/Apollo%E2%80%93Soyuz_Test_Proj...

That brief exposure, before they got their oxygen masks on, put them in the hospital for a couple of weeks. It really could have been worse.

I believe the technical procedure for unprotected ground crew dealing with a leak of this stuff is, roughly, "turn around and run for your life." Hence the guys in funny-looking suits.

The combustion byproducts are toxic and some uncombusted propellants are left in the pipes and can leak out. Has to be washed.

The emergency APU on F-16 fighters uses hydrazine and if it was used and the jet landed on a civilian airport (e.g. engine failure) then you need to have it safely washed before anyone can approach the plane. https://youtu.be/Puia_yQxir8?t=178

You hope it's all combusted perfectly, but there's no reason to take the risk based on a hope. And the combustion byproducts aren't that much less toxic than the fuel. Also not all of the fuel will be used, so there's some in the tanks. That needs to be removed, and could leak if there was any damage.