A longer quote rather puts the "scum of the Earth" thing in context:

A French army is composed very differently from ours. The conscription calls out a share of every class — no matter whether your son or my son — all must march; but our friends — I may say it in this room — are the very scum of the earth. People talk of their enlisting from their fine military feeling — all stuff — no such thing. Some of our men enlist from having got bastard children — some for minor offences — many more for drink; but you can hardly conceive such a set brought together, and it really is wonderful that we should have made them the fine fellows they are.