He said Indians will rarely, if ever, resort to violence. As a foreigner, you never have to worry about being robbed, or having a knife pulled on you, or getting beaten up by a gang of thugs and having your kidney carved out of you. And this is true.
Is not true.
It would possibly be true if he changed it to read "As a foreign man.."
A female friend of mine just returned from six weeks in India a few weeks ago. I'm fairly certain that if Sanjay had met her, travelling alone as OP was, he would've told a very different story. He would've said, as my friend heard from Indians over and over, "Leave. Now. Get on a plane and go back to the U.S. You are not safe here."
She was lucky, and only suffered gropings, attempted kidnapping, and attempted break-ins to her hotel room. But violence against foreign women is on the rise in India. The U.S. Bureau of Consolate Affairs cautions women not to travel to India alone[1]. And "alone" in this case means "in a party without men".
Of course, none of this would be visible to you as a foreign man... you just get treated completely differently. But it's dangerous to spread the idea that women can just go to India and "not have to worry" about violence.
[1] http://travel.state.gov/travel/cis_pa_tw/cis/cis_1139.html