I've left two organisations (BBC and Sony) and neither time did I have any problem with my direct manager in fact in both cases I liked them although I did have a lack of faith the top management and the direction of at least my area of the organisation.

In the BBC case (amongst other issues) my department was earmarked to be moved to Manchester (about 200 miles away) in about 4 years. I was clear (for family reasons) that I wouldn't be moving so staying would have felt like a personal dead end to me (although later the BBC's plan changed and much of the department moved into London instead which might have been OK but the lack of thought through initial decision was a really bad sign about the senior management).

At Sony it was a general lack of faith that the management had enough capability that Sony could become a profitable, viable, mass market electronics company again. That made me happy to leave to see what I could do on my own and be home to take and collect my son from school.