I agree, I'm mostly advocating caution. Just like if you see a car swerving around in front of you, you might want to slow down and avoid. Sure, they might have just swatted a fly or something, but they also might be drunk.
The issue is that cult interactions are crafted to specifically play on social norms. They are HIGHLY manipulative. I'm not saying that you should assume everyone is a manipulator, but you should at least have a little caution while talking to strangers.
I had a college roommate that invited me out to pizza with him and another roommate. I accepted (what could be more normal) and he drove. It was a fucking "pesticide salesman" pitch. That was the last semester I roomed with them.