They can build up a graph and they'll see your emails when sent to a gmail recipient or from a gmail originator to you.
With a very large fraction of all email now passing through Google's servers you can expect them to be able to piece together the missing bits with high fidelity.