Late 19th century legislator to and about another:
"With a few more brains you could be a halfwit."
"He never opens his mouth without subtracting from the sum of human intelligence."
I wrote them down a couple of years ago, but forgot to note the source.