I’ve been in the exact same situation a year ago, where another manager got mad at me for a delay. It was caused by him approving his entire team to simultaneously go on 3-month holidays when the COVID travel ban ended. Apparently that was my fault for not estimating accurately.
But they are accountable to their stakeholders (clients, upper management, etc). They take your estimate and many others from other people in the team and work those into a delivery plan.
A good manager will know how to manage risks and remove blockers in a way that gives a developer the best chance to work within the estimate. A bad manager will usually have no plan and put all the blame on the developers if things go awry.
Eg: I've seen many scrum teams transforming story ponts into days or vice versa missing the actual purpose.