Do you litter your entire house or allow others to do so?

How you deal with it is make it explicit that memes (or at least off-topic memes) are not allowed (and even excessive usage of on-topic meme is frowned upon), heavily enforce that rule and invite people who want to meme to join a meme server.

At least that's how it's done in a ~10K member server and it has worked well so far. Plus it avoids lowering the discussion level to meme-level.