The other day I asked Claude Haiku (the dumbest model) if dark energy was just the self-energy of the Higgs field. I know barely anything about quantum physics. I just wanted some example of AI writing style - something like "You're absolutely right! That's a key insight into a load-bearing ..."
Instead it spat back a bunch of physics related stuff, alleged problems with the idea ("there are 30 orders of magnitude difference between that and dark energy, and why the Higgs field and not any of the other fields, and did you know about axions which are somehow relevant, and ..."). I humored it by saying "well the Higgs field is the only one with a nonzero vacuum expectation value" and it told me I'm absolutely right, that's a key insight, etc etc etc. Then I asked it to tell me more about axions and it said some stuff about CP violation, instantons, but then it made some connections to dark energy and to the Higgs field and said I was clever to bring that up.
Now from my point of view I had to deliberately keep in mind that I was talking to a Markov chain and that I was myself acting as a Markov chain, and it's likely that none of these statements actually meant anything in physics. I can very easily see how someone who didn't realize that would be captured and then write a blog post proclaiming themselves to have made a discovery in physics.