> am I committing violence against him? Should I be force to work for him to avoid that "violence"? I don't think so.

Being forced to do work is violence yes? Slavery precisely. Why can’t you be forced to work? Because you’re protected against it, only because you have the capacity for violence, or at the very least the benefit of promised violence. Otherwise you could be coerced to do anything as you would have no recourse.

We have laws that protect us from coercion, both physical and otherwise. Those laws only exist because of promises of retaliation if they are not adhered to. So yes, refusing labor is a violent act (or at the very least, violence by proxy).

Perhaps I should say “refusing labor with the expectation of not being assaulted or killed, is a violent act”.