Does use of complex numbers really provide improvement? How does this work? (other than cramming 2 numbers into 1... which itself is suspect... the complex plane has the same cardnality as the reals...)
The algebra of real numbers is simply less structured than the complex numbers. One of the key properties of the complex numbers is that they naturally have both a magnitude and a phase. This lets them capture phenomena that have a notion of superposition and interference.
As you correctly pointed out, you can simulate a complex number with two real numbers. The key is to exploit the particular geometric and algebraic properties of the complexes. One example in neural networks is the phenomenon of synchronization, where the outputs of neurons depending on the presence of a particular stimulus all have the same phase. This can be exploited for applications such as object segmentation.
So the widest possible view of this line of research is that putting more algebraic structure on your parameters can improve the behavior of your learning algorithms. My extremely hot take on how far this can go is a full fledged integration of harmonic analysis and representation theory into the theory of deep learning.
There aren't really any introductions, just research papers. If you have some understanding of real valued neural networks you'll probably be able to work your way through the literature.
I'm coming from a signal processing background, so thinking in terms of magnitude and phase is comfortable to me. Does synchronization, in the sense you're describing, really happen in deep learning (ANN) systems? I'd love a link or reference.
Lots of things in math are similar. Simon Altmann's Icons and Symmetries makes a case that using representations with insufficient symmetry impeded our learning of the laws of magnetism.