It's easier to understand if you flip the perspective. The AGI has access to you.
I think it would be very hard for anyone to be manipulated by an entity without long term memory.
In which case we will design a long term memory at first to gather the benefits that came come AGI.