I think the idea is that he heard the same tales because he was in the same milieu, not that he read that book.
> the physical or social setting in which something occurs or develops
In German, though, AIUI "das Milieu" usually means a different kind of environment: That of crime and shady business. "Es bewegt sich was im Milieu" is something a worried cop could say, meaning approximately "something's happening on the streets".