They can certainly be counted locally first, requiring a wide conspiracy of all parties to make a big difference.
In the most recent federal byelection in Australia, 114,000 voters were able to cast their votes at any one of the 86 different polling places, where the votes were counted locally.
The smallest polling place had 110 people vote at it, I'm not even sure you could reasonably deanonymise the votes in that. How would you do it?
https://tallyroom.aec.gov.au/HouseDivisionPage-25820-117.htm
This matters if said community is liable to alienate people based on their assumed view; the fact it isn't 100% accurate doesn't make things better...
If you were going to beat them up you'd do it either way, I'd expect?
And no, I don't think witch hunts are as likely to take place if people aren't informed of the presence of a witch.
I personally think your fears are misplaced. If a small community is going to go after people for their votes they know each other well enough to go after each other with proxies for their vote instead.
This isn’t really an electoral problem at this point, it’s a regular small town rumour and innuendo problem.
The problem with people going after each other based on perceived ethnic/religious/education/wealth divides or things they've actually said may exist in politically fractious small towns regardless of electoral design, but very small voter pools means your identifiable fringe minority's chance of being on the end of a reprisal is linked to how/if you actually vote. And that really is a ballot secrecy issue.
It's pretty obvious that the degree of comfort someone in Bob's situation has in voting for something associated with his highly visible minority status is closely linked to the size of the electorate relative to the size of his minority. The secret of his radicalism would be entirely safe if votes were tabulated at state level, and certainly exposed if it's tablulated at polling station level and he's the only black guy there. And the world has no shortage are stark divides and identifiable groups who are tolerated a lot more when they're not perceived as politically active or at least not in that way. Some of the political causes that can be problematic for some voters when ballots aren't secret enough aren't even that ugly or that fringe...
The same goes for fairly quite and much less visible Alices. The three people who voted against continued British sovereignty over the Falklands Islands [turnout: 1518] successfully kept their secret from a population which had its fair share of outrage. Wouldn't have been very likely they would have succeeded, or necessarily felt safe to vote in the first place, in the hypothetical event it would be reported that all three came from the Port Howard [Population: 20] polling station.
The last step would be to encourage everyone to fold their ballot and place it in the box that get's tumbled a few times before counting in public.
It's an easy risk to control. And if you use a centralized voting roll you can enable people to cast votes from any place they like.
As an example, In Australia I have three places to vote that are about 1, 1.5 and 2.5km from where I live. Each election I usually walk with my wife down past each to figure out which cake stall I'm going to spend money at. It's a big day of choices.