A lot of it is pretty gimmicky right now, granted, but that's not where things are going. The logical conclusion this is all heading towards is really all about coordination of device/thing across the household. Convenience is really at the heart of it, which means for a percentage of the population, it won't ever catch on, because as you say, there's nothing WRONG with a physical key for your house.

But we said the same about cars, but how much more convenient is it to just put your hand on the car door handle, and it unlocks and opens for you? How much easier is it to just get into the car and hit a button to start it because the key fob is in your pocket?

They're small improvements of quality of life, but added up, and it makes for a better experience moving through the world.

Security is a HUGE issue of course, and to a large extent we need to figure it out, but there's good progress being made. And IoT devices aren't (as another comment says) just another outdated Linux box. OTA (over the air) updates are a real thing now, and done securely are as safe as updating your desktop machine at home.