AGENTS.md (and friends like CLAUDE.md) should be for robots only, whether a large single file with h2 headers (##) sections, or a directory with separate sections, is a matter of taste. Some software arch/design doc formats support both versions, i.e. see Arc42.
Though, it's much easier and less error-prone to @-mention a separate .md file, rather than a section in a large markdown file.
Smaller files also might be better when you want to focus a coding agent's attention on a specifric thing.
They're also easier to review diffs / PRs.