I wonder, would agricultural labour be backbreacking if you only had to meet your own demand (and owned enough fields for the purpose)? Serious question, I don't know.
I've only had to work in vegetable and fruit gardens belonging to my grandmother, but if it scales, then yes, it would be backbreaking. Only worth it if you enjoy it or otherwise derive self-actualization from it etc.
Yes it is backbreaking or you are trapped by low margins into ever increasing size of fields so you can afford technology not to break your back.
Nature is too erratic to not have episodes of famine if relying only on sustenance farming.