There is evidence that as LLMs increase in scale, their preferences become more coherent, see Hendrycks et al 2025, summarizer at https://www.emergent-values.ai/
A preference is meaningless without consciousness and qualia.
Consider a philosophical doomsday AI: it behaves exactly as if it has a desire to harm you (meaning that it does!), but it doesn't actually want to do that.
Or we can just drop all this sophistry nonsense.
Look, i understand being concerned about robot and self-driving safety. AI is a nebulous term and we don't need to restrict anyone training AIs.