As a layperson, I agree with the OP...The focus on the quality of teachers -- and firing "bad" ones and hiring just the "good" ones -- has always seemed to me to be overemphasized, as it makes for a sexy, easily digestible political debate. Not that good teachers (whatever your definition of "good" is) aren't worth having, but it's doubtful that they alone can have a significant impact on student outcomes...in the way we should be doubtful that the well-behaved, well-equipped cops from a rich crime-free suburb would, when moved to the Detroit PD, would have a significant impact.
I lived with a teacher and my best friend is a teacher, both are young and about my age and who work at impoverished schools, and I've been constantly amazed at how much of their talk is not about how bad the kids are, but how bad the administration is...over things such as playing favorites (among teachers) and squabbles over office space and, of course, having to buy their own supplies and books (some of which is reimbursed at the end of the year). You can chalk some of this dysfunction to the educational hot topics of the day: the power of teachers unions, teacher pay, standardized testing...but the bottom line is that passionate, effective teachers can be nullified by a weak system...in the same way that a great programmer may be ineffective in an engineering environment with poor testing/documentation processes and a terrible office environment.