My thinking was the same for years, but then I learned about Home Assistant. It's a wonderful open source package that integrates with everything. And there's a lot of options for local wired or wireless transports that keeps smart home usable without third party servers or even an internet connection.

In my setup I have some Wi-Fi connected outlets running Tasmota and some light bulbs and thermostats on Zigbee.