I know. Their thought happens at inference time and only at inference time. I don't view that as a serious challenge to their mental capacity because 1) it's not clear why being unable to think continuously is actually a disqualifying condition to consciousness and 2) it is trivial to engineer a system where an LLM is constantly in inference in an internal dialogue, negating the criticism in fact and not just theory. Current LLMs aren't optimised for that, but we already know they could be with Google's million+ context lengths plus doing something like running RAG on a library of summarised previous thoughts.
>They seem to be a mere organ of an possible intelligence.
That's totally possible, LLMs could end up being a complete AI's language centre. I subscribe to GWT and that was the box I initially put LLMs in. That said, I think there's good reason to believe (e.g. Toolformer and derivatives) that an LLM can perform the function of a selector in GWT, which would make it conscious. We should build it and find out.