Yes, what matters is indeed those variations, not the amount of toddlers. If you randomly removed 99% of the toddler population the variations wouldn't be significantly reduced. Hence the sample size you'd need in that situation would still be roughly the same as now. Yes sure, if you removed 99.999999% of the toddler population it would get so small that you'd have appreciably fewer variations in the remaining population, so the population size is not
completely irrelevant, but it's still
overwhelmingly irrelevant in practice.
You do need a representative sample obviously, but that's a different issue.