When measuring these processes, one approach has been to ask participants to provide confidence ratings. Recollection tends to lead to threshold-like, very high confident responses. Familiarity is more graded and continuous. Many then use a dual-process ROC model to identify the recollection and familiarity components, on average, of a person's memory of a memory test (see work by Andy Yonelinas).
This kind of work goes beyond memory, but applied to the general problem of how people judge their confidence in answers.
Its likely been applied to LLMs. A familiarity signal would probably be pretty easy to generate... The recollection kind of component might take some of those introspection type of approaches. For example, these papers, which I have not read, [1] https://arxiv.org/html/2603.17839v1 [2]https://arxiv.org/abs/2603.09250 might be getting at these ideas.
This might be relevant