From my time building ML models (2016-2023) evaluation and interpretation was a large chunk of it. I often used "brute force" interpretation methods but there were others like SHAP [1] and LIME [2]. The other large chunk of ML work focused on feature engineering and data prep. Today in the LLM agentics era this is out the window. There are "evals" but no one does it seriously, and interpretation is often propagated to the consumer.

[1] https://shap.readthedocs.io/en/latest/

[2] https://christophm.github.io/interpretable-ml-book/lime.html