Following the paper, if you end up with a gender balanced workforce, it implies there is surely a bias in one of the variables - the candidate pool (like you say) or the evaluation of a candidate or other related things. However the bias must also reverse to equalize once the balance tips the other way or actually disappear once the desired ratio is achieved.

Edit: it should go without saying that once you hire enough people to dwarf the starting population of the startup + consider employee churn, the bias should disappear within the error margin in the real world. This just follows the original posted results and the paper.