Even if it was under compression against the outside ring, if the teeth were smaller than those of the outside ring, a force could cause them to rattle.
That said, if the difference in tooth counts is big enough, and the number of engaged teeth is high enough, you might get engagement from the teeth at the edges of the engaged surface, even if the inner teeth are a bit out of spec. This would cause some odd wear profiles in the teeth over time, though.