Can you tell us more about the specifics? What rabbit hole did you went into that was so obvious to everyone ("dude, no", "stop, go for a walk") but you that it was bullshit?
(Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))
This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing.
Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.
Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings.
Linear as in for this input you get this output. We've made astounding progress.
Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.
I checked the content, I do not think that it is useless, and I am sure you have learnt a lot. Perhaps get in a rabbit hole about http://CharlieLabs.ai (your project, before people think I am advertising). :P
Spending all weekend on a puzzle or a project at least keeps you in a tight feedback loop with something outside your own skull. ChatGPT offers you a perfect mirror of the inside of your own skull while pretending to be a separate entity. I think this is one reason why it can be both compelling and risky to engage deeply with them: it feels like more than it is. It eliminates a lot of the friction that might take you out of a flow state, but without that friction you can just spin out.
With extended philosophical conversations there is nothing grounding the conversation, nothing to force you to come up short and realize when you've spent hours pursuing something mistaken. It's intellectual empty calories.
Needless to say this is super common when people go down quasi-scientific/spiritual/woo rabbit holes- all this stuff that scientists don't understand must be related! It must all have some underlying logic! But there's not much reason to actually think that, a priori.
One thing that the news stories about people going off the deep end with LLMs is that that basically never share the full transcripts, which is of course their right, but I wonder if it would nevertheless be a useful thing for people to be able to study. On the other hand, they're kind of a roadmap to turning certain people insane, so maybe it's best that they're not widely distributed.
I don't usually believe in "cognitohazards" but if they exist, it seems like we have maybe invented them with these chatbots...
https://gizmodo.com/billionaires-convince-themselves-ai-is-c...
I believe it's actually the opposite!
Anybody armed with this tool and little prior training could learn the difference between a Samsung S11 and the symmetry, take a new configuration from the endless search space that it is, correct for the dozen edge cases like the electron-phonon coupling, and publish. Maybe even pass peer review if they cite the approved sources. No requirement to work out the Lagrangians either, it is also 100% testable once we reach Kardashev-II.
This says more about the sad state of modern theoretical physics than the symbolic gymnastics required to make another theory of everything sound coherent. I'm hoping that this new age of free knowledge chiropractors will change this field for the better.
If that's true, then perhaps AIs would come up with something just by looking at existing observations and "summarizing" them.
Far-fetched, but I try to keep an open mind.