For 2 vs 4, I've heard it recommended to use Fibonacci numbers to help avoid the confusion. 1, 2, 3, 5, 8... The argument being that with small points, usually things are easier to compare. As you get larger points, it is more apparent if something is a 5 vs 8. So a 2 vs 3? Meh, round up and move on to the next story. 2 vs 5? Orly? Let's talk on why these are (un)like similarly sized stories.
However! Hemming and hawing over close point stories irks me, eps. when the story could have been half way done by the time the team negotiates the points. If points are slowing your team down or not aiding discussions about sizing, then your team should change how it operates; the process should be helping. We've changed ours method to 1, 2, or 3 (definitions what those mean to our team in another comment I left here). Pointing for us typically takes under 20 seconds. Everyone agrees? Good; done. Disagreements? Either round up or discuss it and move on.
The end goal is to help the team create as small of stories that are still delivering value to the org. If any of your agile/scrum ceremonies are not adding value, the retro is a good place to discuss how to improve (or remove) processes so that the team can effectively and visibly make progress.
> usually results in a shrug
This is a problem. I'd suggest to repeat voting after having a conversation. And like I've said before - developers should try to commit themselves to deliver a chunk of work. It's hard to commit to random estimate that does not take your point of view into account.
You could also try replacing story points with t-shirt sizes. If you cannot agree whether the story is 2 or 4 points, maybe it would be easier to agree between M or L. I am not a big fan of this method but I've seen teams for whom it worked better than story points.