Google does research on both their own and other people's models all the time. Here's the blog post and paper if you want to know more:
https://not-just-memorization.github.io/extracting-training-...You appear to be refuting a slightly different point, though. When OpenAI was making Dall-E 2, they found that duplicates in the training set would incentivize memorization of specific images, as you said. This is more like finding a secret cheat code that would make the model regurgitate everything it had remembered, regardless of how "incentivized" it is to do so or even if it had been aligned to not do that.
My personal argument is this: the primary metric that the training process attempts to minimize is perplexity. This is how good the model is at guessing the training set data. Base models are specifically being designed to compress huge amounts of text and we just so happen to accidentally get a decent word calculator out of it. The alignment fine-tuning that happens later adjusts the model to prefer answering questions, but the underlying memorized data is still there.
>The way these algorithms work is public knowledge, there's not really any black boxes in the hands of OpenAI that would be relevant here.
Nope. Modern GPT is entirely a black box. OpenAI stopped publishing model weights the moment Elon Musk stopped writing the checks. Hell, they don't even publish the model architecture anymore. How GPT-4 works, even on a basic "this is how many transformer layers and attention heads we're using" basis, is a trade secret.
Even if you have model weights, the actual meaning of the learned model parameters has never been known; there is active research on figuring them out. One particular problem is polysemanticity. If you look inside a particular hidden layer, you won't see a single "dog" or "cat" neuron in its hidden layers. You'll have a 512-dimension concept bouillabaisse with "dog", "cat", "bird", "guinea pig", "kangaroo", and so on all floating around whatever shape made sense at training time (even if it implies absurdities like "desk is the opposite of loin cloth"). To untangle this, you have to train another AI to pick out monosemantic clusters of neurons that can then be inspected, and that requires extreme amounts of GPU resources.