I would guess even operating the RCS system could launch some of those loose looking rocks to escape velocity. Imagine blowing on a rock that's been in place for thousands of years and watching it drift into space.