Video, all told, isn't that old. We had a society before capturing video. We will have a society after video is trustworthy. The brief window of human history of trustable video is coming to an end. We will get used to it, and be fine.
In other words, it will happen so the best thing we can do is acknowledge this and prepare for it.
Check out the "CaptainDisillusion" channel on youtube, it's full of examples of people deceiving others using video editing software. To my knowledge none of the examples he's talked about have used face swapping.
I mean they can accept just about any kind of special effect in a movie as being possible, but editing videos to swap faces is a huge stretch?
I do believe that some people will fall for them, but the damage won't be any worse than when people fall for shopped images and phishing emails. It's just something we have to deal with.