Thus the GPU is storing a 110M model in gigabytes of external RAM, and paying the power penalty associated with the excess capacity, while the chosen model 110M fits neatly within the FPGA's on chip RAM, and the design can trim all that overhead accordingly.
A more fair comparison would either run a larger model that had both systems hitting external RAM, or they would compare power/performance against some sort of inference ASIC that had all the RAM on chip (maybe a cerebrus, but scaled according to the portion of the wafer actually used for the model).
That being said, it's neat that they open sourced their work, and it's worth looking down this path more.