Fully reproducible model training might simply not be possible if information from the training environment is not captured. In addition to data and code you might have additional uncertainty from:
- pseudo/true random number generator and initialization
- certain speculative optimizations associated with training environments (distributed)
- Speculative optimizations associated with model compression
- Image decompression algorithm mismatch (basically this is library versioning)
- ....things I'm forgetting...
It's just a lot of things to remember to capture, communicate, and reproduce.