I set up all my non-standard lighting fixtures with Lutron Caseta switches, and replaced all the standard bulbs with Philips Hue bulbs. They've both been working flawlessly, together, with my 3 HomePod Minis for about a year and a half now.
It really is a brand thing. Most are bad.
[edit] > My favorite aspect is that the pico switches can be programmed to switch whole rooms of lights on and off...
This is my only gripe with the Lutron hardware. The Picos can only be used to trigger Lutron devices. They don't integrate into HomeKit like the switches do.
This meant I had to go with a bunch of Hue Smart Buttons, which do, but have a tendency of falling out of their homes and onto the ground.
Was looking at the Hue compatible RunLessWire stuff though. [1]