My job involves creating predictive modeling for the VA hospital system. I have recently, in the past month, worked on models predicting the probability of death in the next year, probability of receiving Social work services, probability of receiving a screener which indicates food insecurity, and more. The idea behind all of these is to take our clinical intuitions and the intuitions based on prior research, then gather variables that allow us to use our theoretical intuition to predict future health outcomes. These predictive models may turn into dashboards, which are basically daily or weekly tables that show clinicians which of their patients are predicted to get certain outcomes.
Now, how does this all circle back to biases and disparities? All of our models include racial, ethnic, gender, rurality, age, and many other categories of sociodemographic information. However, at every step of this process, there is a human (whether PI, analyst, or clinician in the final step) looking at numbers/variables/values and making decisions.
Thus, I don't think we can truly separate the AI from the human in current healthcare analytics. We do our best to control for disparities and get down to the brass tax, the actual medical information, but there is simply too much human decision-making in the current workflow to truly divorce the "disparity differential" from whatever humans would do on their own sans mathematical modeling.
Overall: our models are mere collaborators, and until we minimize our personal and systemic biases and disparities, we can't hope to use our fancy mathematical models to minimize them for us.