Do a factory reset, then set it up again before doing the Gemini upgrade. This helped me a lot and it works way better now, even compared to the original non Gemini version.
Then I learned there was no way to give the new Wi-Fi password to any of these devices without manually re-pairing them in HomeKit and reconfiguring all scenes and everything else...
Like, that seems like basic functionality Apple should have included in the HomeKit design.
In the end, I gave up and just put the Wi-Fi password back.