I wouldn't call a 50 year old neighborhood with 1,600 Irish and Black residents "not a large group of people". Calling that "not a neighborhood" is also an eyebrow raising population, especially given that Seneca Village had a large chunk of the (extremely rare) Black property owners in New York State.
You have to be careful repeating historical claims of an area being a "shantytown", as sometimes that is less a statement of fact and more of a way to justify a negative or prejudiced opinion. Given that the same people who called Seneca Village a "shantytown" also used slurs to described the area and called the residents "debased" and "wretched", there is reason to doubt the impartiality of those calling it a "shantytown".