It is entirely plausible that, as you say, they could be piggybacking off of human work internally and using AI like the punching bag it is - dumb but useful. I wouldn't bet against that, but usually it is difficult for people who did the hard intellectual work to have their credit taken away - which might be exactly what we saw with Navier-Stokes drama that unfolded.
> If the models are PhD smart, shouldn't be able to come up with results with minimal guidance?
Smart is born out of wrestling with the problem domain, not in virtue signaling via academic credentials. So anyone leading with an academic credential is a dead giveaway of the PR stunt underneath. Nobody says PhD smart Einstein - because work can lead the speaking and doesn't need a credential as a support crutch.
> AGI and the near super intelligence? It doesn't track.
True ideal intelligence must not hedge, shouldn't pander to popular opinion or consensus. True intelligence must state what is true as maximally true as is correct while being able to derive more true sentences coupled with instant acceptance when proved false. The current status quo with the LLM doesn't display anything even remotely resembling that. Current LLMs confuse truth with popularity, consensus, corporate speak, etc.
My own bet, like all our stack, is on a memory efficient deterministic AI, where the next token is derived instead of predicted. I don't know whether anybody is working on something like that.
The current AI is definitely useful and it would be false to say it has no utility. I just would like everyone and everything to be more honest and true and focus on the LLMs existing capabilities. On how it can be a useful utility.
This hype train of anthropomorphization and grandiose capabilities, which obviously is false, is what pisses me off.