The fact that you can "factorize" 92 into 4 x 20 + 12, means you need less samples, not more. Of course it also requires more and specialized logic in your program, which is the real problem.
Of course, you could just use more developed algorithms to connect words but then why not just use a voice synthesizer base only on recorded syllables?