Yea, I didn't think much about that, but you are right. Even if you train a dataset but use someone else's datasets those datasets could have specific things in them you might not notice.

The hardest part of this problem is the difficulty of auditing. If you use someones open source code you at least have to potential of reading the code to look for something... with a large model that is difficult on a different scale!