Using postgrey[0] fixed this problem for me, a few months back I started getting hammered with spam and now get virtually zero.

[0] https://wiki.centos.org/HowTos/postgrey