If the difference between samples is VERY large, you don't need a very large sample size.

In other words, we're trying to find the chance that the result we got was due to chance. Let's say you have numbers like these:

A: 11, 11, 12, 12, 13, 13, 13, 13, 13, 13, 14, 15 B: 90, 92, 93, 94, 94, 95, 95, 96, 97, 99, 99, 101, 101

What is the chance that those two samples come from the same distribution?

On the other hand, if A averaged something like 12.5 and B averaged something like 12.4 it would require a huge sample size to prove that those two samples come from two different distributions.

Depends on what effect size or power you are looking for and/or other factors like variance in the data (standard deviation), etc. There are some different ways to calculate sample sizes here: https://en.wikipedia.org/wiki/Sample_size_determination

or google: sample size calculator