The issue is that it becomes necessary as marginalized candidates are given preferences during the hiring phase. So if the same privilege is not extended during layoffs it will unfairly impact them.
By unfairly you mean fairly.
Making "improving company diversity" a terminal goal is a problem in and of itself.