Several papers have explored emergent linguistic structure in LMs. Here's some early introductory work in this space. Despite having explicit syntax parses etc as input, models seem to learn something like syntax.
https://arxiv.org/abs/1905.05950
https://arxiv.org/abs/1905.05950