I was interested to see a paper from a philosophical perspective but it seems this one fails to grapple with some of the more foundational critiques, primarily around reference where it claims that externalism militates for a view where reference is purely situated. I read the cited paper (Mandelkern & Linzen) and the arguments seem very weak to me that an intelligent agent "refers" in the absence of any grounding whatsoever beyond relative semantics between text. What it seems to argue is that because we sometimes are able to refer without absolute grounding (due to lack of direct experience of the referent), then reference overall does not require the referrer to have any world-grounding whatsoever. To me that's a leap. I would like to see more working against phenomenology here, especially Brian Cantwell Smith's so-far-unanswered critique of machine learning as intelligence.

One of their other arguments, which is under the umbrella of the "re-description fallacy" hinges on the observation that describing (e.g.) a piano as "hammers hitting strings" doesn't preclude it from more complex behavior like harmony, and so a simple description of LLMs as autocomplete etc doesn't preclude more advanced understanding. While it's true that any complex process can be inappropriately dismissed by simplifying its description, I think this too-neatly sidesteps more complex critiques which advance from simple descriptions of LLM behavior as contrasted with complex behavior from sentient intelligence that cannot be implemented by those simple mechanisms.