They even beat natural gas on operating cost for many installs these days:
Gas Furnace: 1 Therm = 29.3kWh, Cost $1.5 (at least where I live), Furnace efficiency = 95%. Cost per kWh heat delivered = $0.053.
Heat Pump (COP=5, EER=17): 1 kWh = $0.14, COP =5, Cost per kWh heat delivered = $0.028.
Heat pumps are quite a bit more expensive to install though.