A patent holder can charge whatever they want, or even refuse to allow anyone to license it.

The difference is that Qualcomm agreed to FRAND (fair, reasonable, and non-discriminatory) licensing terms in exchange for their patent being included in the standard.

Once they got their patent in the standard, now they're saying they get to charge whatever they want.

That's the problem. That wasn't the deal they and everyone else agreed to, and it's too late to remove their patent from the standard.