The book was probably Liu Cixin's _The Dark Forest_.

If you want to read it, start with _The Three-Body Problem_, the first in the series.