Noise cancelling headphones will cancel low frequency sounds like fans / AC, but not high frequency ones like talking and phones.
The most disturbing frequency band (on cognitive tasks) is probably the middle frequency, especially when the noise has "information" (language, tones, signals); but on a physiological level (directly affecting health (CNS function), performance and well-being it is the LOW frequencies that, albeit not cognitively disturbing at first, will have the greatest effect on a long-term perspective.
Headphones/noise cancelling headphones are good a fixing problems in the mid/mid-high range but perform poorly in the low frequency range.
My suggestion is this: your health comes first, no matter what. If you develop a noise-related disorder, you will under-perform and eventually get fired. I would do two things: use an app like SPLnFFT to document the noise level. Use a spectral analyzer to find out where the noise peaks are. ... then get proper in-ear hearing protection from an audiologist (comes in skin-colored, looks like an hearing aid). No-one can blame you for investing in your health to maintain your ability to perform, right?
Literature: http://www.noiseandhealth.org/text.asp?2001/4/13/33/31803 (K.P. Waye. Low frequency noise pollution interferes with performance)
Do you have a good recommendation for a recording dB meter and recording app suitable for graphing noise levels over a daylong period? Such a thing would be useful in proving the noise load of a busy open office and allow it to be compared to OSHA standards.
It's a recent Sony top-model, which from reviews seems to be on the top together with Bose regarding NC technology at the moment.
They're great for airplanes, trains, and low-frequency background noise, but for a crowded office where people are having chats and telcos, I'd rather use Shure in-ear monitors.
When they are on, but not playing music, I can still hear loud voices but it’s significantly dampened (like in the room next door).
With music playing, at a relatively low volume, I can’t hear anyone - unless someone is shouting right next to me.
This was only slightly paraphrased from written policy at a former workplace.
You have to show some political savvy; wearing headphones in a open office is exactly like being noticed playing games on your phone or sleeping while the CEO speaks at gathering, or making a big production of refusing to answer phone calls or questions from certain coworkers.
There's a critical distinction of scale; if "they" don't want you working and prefer talking and distraction to the level that they bake it into the physical architecture of the office, that's really bad for the company on a large scale, but on a small scale trying civil disobedience by wearing headphones will just get you fired. Personally on the downside you're better off crashing the company than getting fired and on the upside the people responsible for the large scale operation of the company probably have quiet private offices anyway and aren't going to reward you for fixing the company anyway even if by some miracle you did it.
Open offices are literally in the most straightforward sense a declaration the company has no idea what its doing so talking about it is a good first step. Cutting yourself off from that with headphones means they may as well not pay you.
Civil disobedience in this case has no positive outcome; ditch the headphones.
A constant background noise is unpleasant but not so distracting - and it gets filtered out.
Sudden voices are way more distracting and they don't get filtered out. On the contrary, they stand out even more once machine noises are filtered out.