You can use complex numbers number to represent higher dimensional objects using only primitive operations, scalar values, and an imaginary number for each dimension. However, computing with these values is significantly more challenging that real vectors. This book on 'Geometric Algebra' starts to explain: http://www2.montgomerycollege.edu/departments/planet/planet/...

Complex numbers are a particular 2d slice of 2x2 matrices that happen to capture rotation and other periodic phenomena very well. If you are trying to solve some problem that you suspect to involve periodicity, focusing on complex numbers helps you get there faster.

Maybe it would help to think of the turing machine as analogous. Many programming languages are Turing complete, you can express any computation in any of them, but some languages are more expressive than others and let you reach and work with ideas you wouldn't conceive in a less expressive language.

Lots of things in math are similar. Simon Altmann's Icons and Symmetries makes a case that using representations with insufficient symmetry impeded our learning of the laws of magnetism.

They have some pretty useful properties. For example every polynomial of degree n has exactly n complex roots and if a complex-valued function is differentiable wrt a complex variable then it's also infinitely differentiable and analytical.