If you aren't in the right-most lane, then yes, you are wrong. See, e.g., Ind. Code § 35-44.1-2-13; Virg. Code § 46.2-842.1.
But there are drivers who weave through traffic whenever they see an opening and those are the ones who appear to be angriest when they’re behind me in the right-most lane because they were impatient in the other lanes.