Comment three: after hacking my way around that, the script ran for another few hours doing disk IE (writing the offload) such that my directory is now 1.3T (330GB .git/lfs/objects, 330GB sharded weights, and 607G of "offload".

Then it failed again:

    File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/big_modeling.py", line 188, in dispatch_model
      main_device = [d for d in device_map.values() if d not in ["cpu", "disk"] [0]
  IndexError: list index out of range

Now I'm curious just how long it will take to repro this error (IE, running again, with the offload files aalready written). It's also puzzling since I set the device_map to 'auto'.

Again, all par for the course and stuff I expected (having worked in HPC/ML/science for 3 decades, you get used to research codes).