Historically researchers looked at activation functions that are at least somewhat justified by nature, like the sigmoid. More recently people have focused on the mathematical aspects and figured out that the rectified liner unit `ReLU(x) = max(x, 0)` works just as well if not better for most applications.
Maxout networks (https://arxiv.org/pdf/1302.4389.pdf) are another interesting development in that direction.
There's still lots we can learn from biology, of course; but the last decades rapid advances we saw from deep learning at most barely relate to real brains.