Yeah, I'm pretty sure you're right. I've asked a dozen or so questions, of for every one of them I've gotten an answer that seems related to to the words in the question, but not in any logical way:

Q: What's the difference between a proton and a neutron?

A: Atoms are made of protons, neutrons, and electrons.

Q: What trajectory do planetary orbits follow?

A: Kepler's laws of planetary motion describe the orbits of objects about the Sun.

Q: How do you measure the charge of an electron?

A: Electrons have negative charge.

Q: What conservation law is the result of the time invariance of physics?

A: As a result, the law of conservation of energy has been changed into the Law of Conservation of Matter and Energy.

None with high confidence of course, but it gives you a sense of roughly the sort of 'reasoning' this thing is doing.

Indeed, I did several questions all intended to be simple variations of the main examples. It did not give a coherent-sounding to any of them.

But it looks like responds to the example with full paragraphs. Maybe it's real but coherent 10% of the time and they recorded the questions that yield coherent answers.