Probably power, getting the data onto the FPGA, and utilising FPGA's being unlike software.
I definitely want one but any common task worth having on an FPGA is probably common enough to justify either a GPU or actual silicon.
Intel and AMD both have the IP to do it, and iPhones do have a Lattice chip on them apparently