My impression is that chivalry was more about relations between nobles than anything to do with peasants.

Feudal lords have pretty much always taken whatever liberties they like with the local people: rape, theft, property destruction, murder, torture, kidnapping, ...

Medieval knights were little different than the thug soldiers of any modern criminal gang, except less checked by state power.