That's not how fair use works. This kind of use would facially fail three of the factors for fair use; it's not be transformative, it copies the original work in its entirety, and it harms the commercial market for the original work.
Fair use doesn't enter into it if weights aren't copyrightable. They're machine generated by stochastic gradient descent. There's no human hand setting the weights. We won't know until something ends up before the USCO.
how much do the weights have to change for it to be transformative? Edit: you arrive at the same question as taking an image with img2img and running it through a diffusion variant - keeping the composition, colors etc. but no the details
Much of the case law about the "transformative" factor focuses on "new meaning or expression", but it's about visual art, which is generally very difficult to reason about w.r.t. copyright. I think the example to look to for technology is Authors Guild v. Google, where "transformative" is more about non-expressive purpose, and it was considered transformative to copy a bunch of books to produce a search functionality, since the search functionality (which only displayed snippets) was a transformative purpose compared to the underlying creative expression in the books.