Yep, reading something doesn't invalidate your caches. Or everyone else's caches.
In high performance applications, you must not let the kernel choose randomly what pages should be invalidated, but you must use madvise for memory_mapped files or posix_fadvise for files accessed with read/write, with appropriate arguments so that cache memory pages that are not needed any more will be freed with enough time in advance before reading more data.
You'll notice that removing entries from the cache causes exactly the same effects as changing the data that backs them.