I have never worked a real farmers day of work in my life. I sit at desks and use computers for up to 8 hours a day. I have never worked a field, built a house, or done anything significantly physical for work ever. This is not unusual.
If they believe you are not doing real work, what it tells you about the fact they still supervise you? They are admitting that they are useless at their position and that trained pigeon would outperform them.