Don't you find transformer based models perform better? Are they too heavy?
Given that, we have trained a variety of models and we are still experimenting on what works the best. We are even considering using VLMs for certain tasks if we can fine-tune them well enough.