Explainability for an NLP model is well within reach. I predict OpenAI will have something like this in the next 1-2 years