I found that trying to remain on good terms with my line manager and director helped feel less guilty about moving on. I guess it's the thought that the door is not fully closed.