There are though absolutely places with a large black population which have serious crime issues, but you see similar crime rates in impoverished areas that are predominantly white. Calling it a problem in black America makes it seem like a black problem when that is correlative rather than causitive. Poverty is the core.
Historical inertia, past (though fairly recent) laws, etc... are part of a complex story of which the result is poverty among a specific demographic (though not limited to that demographic of course - the extractive mining towns in Appalachian areas created parallel stories of systemic poverty in predominantly white regions).
It takes a long time for societal wounds to heal.
The prompt was whether blacks leaving black neighborhoods would be labeled racist. The assumption is that although it is categorical racism, nobody would call the act racist.
As for crime, it's such a messy topic, though, recheck. I can easily find a lot of studies showing black communities having higher gun homicides, etc. after controlling for wealth (which you disagree with).
The way societal traumas manifest is tied to the types of trauma each demographic experienced and experiences (including their own self-perceptions of the ways in which they have been victimizes).
Poverty is often a stressor that squeezes out behavior we tend to identify as criminal, but it just a common factor in exposing the wounds.
Depending on the group in poverty, it may manifest as gun violence, physical violence without guns, domestic violence, theft, stimulant abuse, opiate abuse, and a myriad of other things.
i.e. if your cultural wound is to feel powerless, a gun may make you feel powerful; in charge.
If the wound is anxiety, you might choose to numb out.
Controlling for wealth only gets you so far because it is a single dimension.
If you want to bring up generational trauma, then it sounds to me like you're making the argument to leave a neighborhood based on skin color. Yet, I don't know how to reconcile that with your criteria that racism is about intent regardless of risk.
The type of crime is the product of complex factors.
The presence of crime in general is also complex but is exposed by and increased by poverty
As for racism and intent, leaving an area because it's unsafe is one thing. Leaving an unsafe area because you think black people are inferior AND because of safety is another thing entirely.
In the same way, it is one thing to understand historic trauma has negatively impacted a demographic, and another thing to decide that the behavior you're disturbed by is intrinsic is another. That's the reason that discrimination based on a trait a person cannot change (like skin colour) is racist even if you argue it is statistically rational. Judging people individually on criteria of character, ability, etc... is a recipe for better social outcomes overall.
I think I would call the act racist because what makes it racist is tied to intent. But one could argue otherwise I suppose. That's just my take.