Because only a finite amount of time passed since the Big Bang, and all speeds are finite, so it could have grown only a finite amount.