This has happened a number of times that I remember - one was a worm/exploit that would patch the hole.

I remember reading about 5 or so years ago that the FBI was doing exactly this to counter the EternalBlue exploit (I might be getting the exploit name wrong)

this was awesome. people are surprising