The biggest reason I'm a skeptic though is the incredibly carefree and reckless approach I've seen used by many IoT devices.
The Nest is a great example - I was shocked to discover there's no division between the part that directly controls the furnace/AC/etc and the rest of the device. That should've been a no-brainer, because first and foremost the device shouldn't cause harm, and when the "smart" portion inevitably has a problem (even if temporary), a user shouldn't wake up to a frozen or overheating home.
This stuff needs a higher level of safety and guarantee, and treating them like smartphone apps is going to end very poorly.