I wondered about this too. But even balloons are maneuverable in some sense, by changing their altitude. So maybe the meaning is more like "didn't maneuver in response to our presence".
Most balloons are not equipped to actively change their bouyancy.