It definitely needs to be left out of anything with non-statisticians in the intended audience. I've started leaving it out of most reports. If I write about a difference, it's statistically significant. The test just gives me confidence to write it.