SymPy (sympy.org) is a well-developed Python package that works with symbolic/analytical mathematics on a level comparable to Mathematica's.

Octave is more of an open source Matlab equivalent. That is, mostly linear algebra.