To me, points should be an estimate of complexity. Higher complexity means more unknowns, more traps, larger possible range of time to complete. If the current codebase is murky, then I add points. I promise that if we fix the murky areas in the code then we can build faster and more predictably.
A 1 pointer means nothing can really go wrong. It's a linear time task.
I don't think the Product Manager digs my philosophical approach.