I'm always worried about LLMs unintentionally affecting the actual content, so the extra effort of carefully reviewing the diff just isn't worth it. Markdown + PanDoc is more sensible to me if your document is simple enough.
Markdown + pandoc is not the same: you need already basic formatting. You have to think about the formatting. For example, math formulas and code are gnarly to type, so I have snippets for them to avoid getting out of the flow.
But LLM don't need that, you can just dump your thoughts, and they format it.