The reason I do maths in my head is that the interface to anything else is still not good enough, or at least my handling of it isn't. Looking and handling a phone takes me out of the situation in a way that feels worse than doing the thing in my head.
If it didn't, if the neural computer interface was real and seamless, I would use that for sure. Come to think of it, I probably wouldn't even notice.