Would that just be the ability to chat without making new memories while using existing memories?

In chatgpt at least if you start a temporary chat it does not have access to memories.

Well you're in luck! They have that feature and talk about it in the article