ACT worked well and my repo is here: http://github.com/trzy/robot-arm

Re: LLM for supervising an imitation learning policy, check out the recent paper “Robot Utility Models” :) They even bought a TLD for it: https://robotutilitymodels.com/