So like you said, devs learn to get better at forcing the PM's to split the stories into smaller chunks. Smaller stories are almost always easier to accurately point since it is always 2 (1 point for the story, 1 point for testing).
The numbers are not meaningless though. That is why velocity is an average of 3 weeks. A single week doesn't blow things out of the water and if it is a new project, the expectation is that we won't know where we are at all those first 3 weeks. It gives a built in ramp-up period for the project.
Agreed, the PM's role is SUPER important. For this reason, at Pivotal the PM's usually have a developer background.