The least you could do is acknowledge to the person behind you that you're about to do it. You're acting as if you're being told to file a permit. On the contrary, it's pretty common courtesy.
I've been flying for the last 15 years regularly and never once witnessed someone forewarning the passenger behind they were clear to recline their chair.