You'll probably hit memory limits if you go much beyond 512x512-sized holes.
Additionally, computation times grow quickly with higher resolution and you already need a high end GPU for this resolution to get a reasonably interactive response time.
You'll also need a favorably licensed pretrained model or a few 10000 training images and masks.
So all in all, I can't see any deal breakers, but I'd probably still use PatchMatch instead.