It seems to have been Thomas Hart Benton speaking about John C. Calhoun:

http://books.google.com/books?id=JLpYAAAAMAAJ&pg=PA34...

And you actually got the quote pretty close to correct:

"No, sir. When God Almighty lays his hand upon a man, sir, I take mine off, sir."