Agency (short or long term) is still in the realm of scifi. The scary part is bad humans manipulating these systems. The AI waking up and becoming skynet is not a realistic threat in the here and now. 4chan screwing with your ai for the lulz, is.
I respect the rationalist argument if there's a 1% chance to achieve AGI, we should spend 1% of our resources to think about it. Makes sense! But having it come up in every discussion with just speculative opinions doesn't really advance the cause, and gets tiring soon (to me, at least).