Aren't asteroids very varied in terms of compactness? Aren't some asteroids piles of rubble and some others really hard rocks? Maybe I am confused but I'd guess this would also play a role?
Chemically bonded objects e.g. solid rock or a huge diamond could approach closer to the planet without breaking up due to tidal forces, depending on the bulk material strength and melting point.