It doesn't seem like this is a problem with self-esteem. The problem is that American schools don't have good educational standards or trained teachers. In the countries that rank ahead of us in the international assessments, even elementary school teachers need to have education degrees. That is not the case in most American school districts (just need a teaching certificate). Teaching is also a more highly respected profession in other countries.

It's pretty much expected in the US that kids don't really learn much of anything in school until they get to 8th grade. I certainly think we could introduce more advanced math concepts at earlier grades instead of taking five years to cover basic arithmetic.

That's not to say that I would like us to adopt a system like that of China or South Korea, in which students are drilled from morning to night on mostly rote tasks and then pitted against each other in a gladiatorial competition called the National College Entrance Examination. If there was a better way of squashing children's natural curiosity and love of learning, I can't really envision it.