I think there should be a painted line, like they do have in some high-traffic areas in Asia, to show where the "opposite lane" is.
Of course, the other thing that really irritates me is when I'm moving in one direction along with everyone else who is also keeping to the right (replace with left if you're reading this from a "keep left" country), and encounter someone head-on, going the completely wrong way counter to the flow. I usually gesture to such people by pointing left, and refuse to let them pass on my right.
Fortunately people are mostly a bit more organised when they're driving...
If more people are walking in one direction than the other, more space is naturally allocated to them.
If you're "merging" from the left, you can pass on that side until you can get to the right instead of waiting to do a left turn, often without slowing anyone down.
Rigid rules are good at stopping catastrophic collisions, they aren't so good at optimizing use of a complex resource like a sidewalk. Consider how things like telephone polls make some areas of the sidewalk 'less valuable' and natural passing/turning/collector lanes.
Ideally everyone should just stand in the middle.
In extremely congested areas people will stand on both sides, but the convention is to stand on one side and have a "fast lane" on the other side. I realize it has less throughput, and puts more strain on the escalators, but that's not changing any time soon.
The point is, escalators and perhaps train boarding are the only two times where people will be mostly self aware of their position. Walking on the sidewalks is not one of those times, everyone is staring at their phones, or walking in groups of 2-3 across.
Since then, I practice relaxation techniques when walking in the streets, and I'm not bothered anymore.
I've just got to stop and walk around the other side of the person I'm with when they start doing that.
Its incredibly tiring being hyper-aware of your surroundings like this, always trying to avoid any awkwardness.