I don't think that argument has a lot of weight. By the same logic a zip file of a book doesn't resemble the original work if you look at the raw bytes but you can extract it and get it back. It would be hard to argue that the copyright violation happens only when you extract the zip, and not, say, if you distribute the archive.
To use, and invert the example from the other commenter, you can even get an existing poem out of a generative model that genuinely was not in the training data at all. This is because it's not an archive, which does correspond directly to one particular work, but a generative technology.
Just because the models are not perceptually the same as their training data does not mean that they are not effectively a lossy compression of that data.
You might get replicas of incredibly popular works like Mona Lisa due to overfitting but that's it.
If I am wrong please do provide an example as this is very relevant and interesting (and impossible from an information theory pov).
The approach I've seen is to prompt for people with unusual names, they're often be only a single source image in the input data set that gets reproduced by the AI.
I've seen examples with the AI "generated" images and the source image side by side - I'll try and find them.
EDIT: Link to the paper https://arxiv.org/abs/2301.13188 and this articles has the example images: https://www.theregister.com/2023/02/06/uh_oh_attackers_can_e... Look at the Ann Graham Lotz image and tell me that isn't the source image being reproduced in a lossy manner.
Wow, that changes everything. BRB, I got books to pirate.