You're absolutely right about ACT. What are your thoughts on diffusion policy line of work? In my personal experience, I found it way more robust than ACT. Have you had a chance to try it?
It's been interesting, for sure! More robust than ACT as long as you are not using action chunking (has the same problem as ACT). Downside is, it can be too slow/require too much data to train. I have a sneaking suspicion that at that scale of data even visual nearest neighbor stuff would be similarly robust.
Of course, no one has figured out an iota of useful generalization, which is sad across the field.