Yep, no doubt there are definitely issues with inflation. My main point is that the size of black holes in the early universe is largely orthogonal to the whole big bang/inflation debate. Even if the model changes, the lack of IMBH evidence means that any universal history model must have some kind of time asymmetry.