The downside is it slows filesystem access down across the board until the caches get repopulated. So sure, go ahead and use it if you like having a slow filesystem.

Caches exist for a reason. Deleting them willy-nilly tends to be a bad idea.

Yeah, I've had zero problems doing this for over a year. I used to get into swap hell every other day and now my life is beautiful and awesome.