Great work, well presented too.

In my experience working on applied DL research problems the training bottleneck is almost always the data loading. This is because datasets for real-world problems usually don't fit in GPU memory (not even close) and often require expensive pre-processing. The latter can obviously be driven down using similar methods to here, but the former seems insurmountable. It's not clear to me what the path is to getting these workloads down into this rapid experimental iteration regime.