Mathematica does so much more than Octave that it's not even remotely comparable. In terms of features, Octave inherited Matlab's matrix-centric mindset by having the \ operator solve linear systems whereas in Mathematica it's slightly longer (LinearSolve[m,b] or just Inverse[m].b for non-numeric ones). But Mathematica offers symbolic computing as well, and you can very easily go from "solving this equation with machine-precision numbers" to "solve this equation with arbitrary precision algebraic numbers" to "solving this equation with an unknown parameter with conditions" in a seamless way. There are whole areas of mathematics, pure and applied, that can be done in Mathematica but not in Octave. In terms of ease of use, Mathematica also excels here; both data and code share a uniform notation that is very similar to lisp, and functional programming ideas carry over very seamlessly.

I'd say if you are someone who's even remotely mathematically minded, learning to use Mathematica is an extremely useful experience. As an example that recently happened to me, while doing front end work I needed to construct certain geometric figures as an SVG polygon; I used Mathematica's built in graphics primitives combined with its plotting capabilities to find the polygon in no time. Even though the end result is a piece of JavaScript, Mathematica gives me a way to prototype the whole thing in an environment that's "batteries included" and well-polished.