What was extremely hard, is havigation and control to enable return back to launch site optimally, going through narrow corridors of acceptable parameters (go too shallow and you risk being unable to correct your position in the end as precision drops, go too steep and rocket breaks up due to aerodynamic forces, and so on), while burning minimum fuel. That was an insanely difficult GNC task which took so many iterations to perfect out...
Plus yes, on landing they didn't have a second chance as rocket couldn't hover - minimum engine thrust was bigger than it's mass so if it missed, it either crashed, or flew back up until running out of fuel and crashing, too.