You are objecting to using the word 'slave' as a descriptor, except in extreme cases.
But the term slave is often used in culture for a lot of things.
Being a 'slave to your addictions', 'a slave to your desires', 'an office slave', 'wage slave'. etc.... I'll spare you a list of hundreds of examples.
I think since you are the one objecting, then you draw the line.
I tend to think it does hinge on 'agency'. If you are trapped. But, sure, being trapped in an office today, is not as painful as being trapped as a slave in the US south. Does that mean the office worker should just shut up and deal with it, it is comfortable enough?