Mildly curious how many full experiment iterations they ran to get to those numbers. Also curious if it would have been possible to run the entire computation on a single node with maybe slightly higher ram, like 128GB. That’s the claim to Dask, but every time I try to use it, I trip over my feet.
If a year of data is 6TB, you could even keep the entire thing in RAM with a memory optimized instance.