In your grandparent's situation, they worked the fields because nature took away their freedom, they didn't have a choice about whether or not to work the fields (unless you count dying as a choice, but most people don't). There may not have ever been a person coming and holding a gun to their head and telling them to plow the field, but at the end of the day their situation was the same.
To someone being forced into an abusive job, it doesn't really matter to them why they're being forced into that job. If someone is blackmailing them or threatening them to do that job, then they don't really have a free choice. If a government has socialized the entire economy and is forcing them to work that job under threats of imprisonment, that's not really a free choice either. And in the same way, if the economy gives them no mechanism to move away or improve their life, and if they have no choice except to keep working or to starve, then they are still being forced to work. Their choice is still exactly the same as it would be under any other coercive system.
Natural systems can be coercive, there doesn't need to be a human deliberately forcing the system to work that way. It's about whether or not someone can realistically make a choice.
To me, it is very strange to describe a situation where a worker has no real leverage to choose to do something different, and to describe that person as being free -- regardless of the scenario that has put them into that position. This gets back to a previous criticism, which is that "nature" is very often neither fair nor free. If there is any major praise of collectivism, it's that humans decided to build systems that were more fair and more free than nature would be on its own.