Both Maxima and GNU Octave are pretty old (mature?) projects with probably outdated-looking (feeling) GUIs. Just to mention some younger and more "fresh" projects: In the SciPy ecosystem, there is SymPy
https://www.sympy.org which has the obvious advantage of being Python, gaining from a big (also scientific) ecosystem, such as numpy, pandas or the JupyterLab.
A large OSS player is nowadays also SageMath https://www.sagemath.org/ which is a kind-of-meta CAS which can actually proxy evaluations to Mathematica, Maxima, Octave and many others, in a more or less uniform pythonic notation.
Probably needless to say, when it comes to evaluation power (for instance for integral solving), Mathematica is unrivalled from OSS projects. However, in certain fields (such as number theory for SageMath or the overall web notebook infrastructure by the IPython and Jupyter projects), the concurrency is catching up or may already have overtaken Wolfram's products.