They did that and the result was their stuff was burned to the ground. Afterwards, white people in power tried to pass more stringent building codes nominally on the theory that "They burned to the ground because they weren't built very well." completely ignoring the role that actively hostile racism played.
It didn't "just burn to the ground" due to some accident. It was torched by an invading force.
And that invading force then tried to change the building codes to prevent it from being rebuilt while pretending "it's for your own good -- your stuff burned to the ground because you losers didn't build it well enough."
The same applies to a lot of social justice issues. To some degree, it's helpful to just focus on building your own dream, but there are some very real issues where that's really not sufficient because other people will actively seek to destroy your dream and/or you because of who you are.
This needs to change. This unfortunate truth needs to be something much more in the past than it really is. (There are people who like to claim it is in the past. It's really not and such claims are part of the problem.)