Were there any other significant factors specific to his business that made you mention customer disputes as such a significant reason?
(Note that I'm not asking what the factors were, I'm just asking if there simply existed any.)
The reason I ask is, I imagine there might have been other similar businesses with a high rate of customer disputes, and I imagine there may have been other factors in his business that could have resulted in you not wanting to work with them, both of which would be reasonable justifications for avoiding this business -- but what baffles me is how a company with "zero customer disputes over [their] entire history" can get an email blaming his customer dispute risk as the primary concern.
Note that I'm not disagreeing with the soundness of your ultimate decision here, I'm just confused by the email you sent justifying it.