The time aspect cannot be emphasized enough. Time is a huge issue for adult language learners. You simply need to spend a lot of time listening, speaking and forming relationships in the target language.
As a child, socializing and learning is your full time job, so it's a lot more straightforward. No pesky job or dependents to take care of.
Another issue people don't discuss much is personality / identity. As a child, we are so much more malleable with regards to that. It can be very difficult for adults to fully step back out of their cultural and social sense of self and enter a new context where they are totally clueless and inept.
As a child getting set back a few years temporarily isn't a big deal—you just get put in a few ESL classes and you can still play games and socialize with your peers. But to go from a high functioning adult to a toddler is quite taxing emotionally.