Yes, during training multiple checkpoints are created, you can distill from any checkpoint you want.