> So then, why did the size of armies grow? If a smaller armored force could route a much larger unarmored force in spite of firearms... why didn't more armies stay smaller and armored?

Because of societal changes, as governments became more centralised and central authorities (usually monarchs) didn't want to rely on aristocracy, who could easily challenge their power, but preferred the masses, who were easier to control as bureaucratic and propaganda apparatus of the state grew more organised and powerful.