This seems like a point in favor of models like REALM (https://ai.googleblog.com/2020/08/realm-integrating-retrieva...) which could allow for deletion of sensitive information without needing to retrain the model.