To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
Inference is where it falls short really, and solutions like vLLM are much much faster.