Does it have to be open source? Because LibreOffice is a mess, especially on macOS. Apparently, something called NeoOffice is LibreOffice for macOS, and while sold commercially, is GPL.
Complex documents (long reports, mathematics notes, ...) => LaTeX. (TexMaker for example).
It is totally worth it to spend 1 or 2 days struggling with LaTeX.