'In fact, they are probably not redistributable unless all the training data is supplied, since the GPL's definition of "source code" is the "preferred form for modification". For a pretrained neural network that is the training data'.

To me, this is the crux of the matter. If the view is taken that software distributed under a "free-software" compatible license is non-free without the ability to obtain all training sets for any models, there's going to be huge difficulty in incorporating ML into free software in many cases. Since many datasets aren't able to be redistributed freely (e.g. licensing, legal, cost), that's an enormous advantage for non-free offerings.

A possible route around this might be 'community curated' datasets, where contributors freely license their data in the same way as we do code. It'd be interesting to see if they come with an analog of the AGPL - ie. models trained on this dataset must be released to the user as source. (This might already exist?)