I am not a lawyer, but my understanding is that plant breeders who develop a new cultivar or variety are generally awarded seven years of exclusivity on registration, which seems reasonable to me. For fruit trees that take 3-6 years to become productive, you could make a good argument for perhaps doubling that time.

In fact, I think a lot of IP law would be better if it resembled this arrangement more closely, in that it would incentivise the actual author, developer, inventor, etc to be rewarded for their work without letting people who had no connection to that work to collect rent for decades or centuries thereafter, long after those people have moved on or died. Walt's grandkids should not still be collecting rents directly from his work.

Extending these rights on plants via patent law is ridiculous on its face to me for all the reasons you've already listed, and should be thwarted at every turn. Using a tool that someone else invented to tweak a species that has existed since before civilisation, and which has been improved throughout civilisation is no more a new invention than changing your irrigation and fertilisation timing to improve yield. It doesn't deserve a patent.