I use memory all the time so that I can ask it about things. That doesn't mean it "knows me" any more than my notes documents do, it's just more convenient to search.
But are you happy handing over your notes documents to mass aggregators of information?