Sound in general makes it so hard to concentrate; human voices are worse. I suspect brains are built to pay attention to human voices, or at least mine seems to do so. Noise attenuating headphones don't help; they work ok at blocking out trucks but poorly on voices. Earbuds plus music do an ok job, but the studies I've seen show decreased performance when listening to music. Even if you think you aren't, part of your brain is paying attention. And really, why do I have to listen to music all day to drown out the noise of people.
And let's not forget visual distractions -- all the shit waving around in my field of vision doesn't help either.
Like I suspect many of you, I'm consistently far more productive late at night. In my case, that's because it's the only time I have where it's quiet and I'm left alone.