Dropout is only used during training a neural network and also neurons are turned off not removed.