>The problem of preventing leaks of forbidden knowledge seems impossible without excluding the forbidden knowledge from the training data. I fail to see how that would be possible without crippling the desirable use cases.

This isnt accurate. There already several tools and methods explicitly for excluding forbidden knowledge from training data, or reweighting the training data so that the LLM have the perspective that the owners want.

e.g. if you want the model from being racist, you run the training data through a classifier to remove data with negative sentiment around race.

https://openai.com/research/dall-e-2-pre-training-mitigation...