The issue is that leaving food for the animals alters their behavior and can cause issues. You're probably not the worst offender wherever you are, as there are many people who just give food to animals, but the argument would be that it contributes to the problem.

Actually banana peels and orange peels don't biodegrade that fast in a north American forest.

Apple cores I think might not be too bad to toss, ants can eat that up in a day or two.

I pack out the peels of any tropical fruit.

Please don't do that with banana or orange peels, they remain present for a long time. I have no problem personally with people tossing apple cores in more remote, little-visited areas (though it still makes sense to crack down on such things a bit more in heavily visited areas like National Parks) but various peels don't rot well and remain in place.

I don't have a link though, it's just something that was been repeated at length by the Natural Reserve agents and the ecobiologists to the personnel and visitors while I was on Kerguelen island (precisely about apple cores and egg shells - most of the vegetation is actually in the same Rosaceae family as apple trees, and there are many local bird species).

It makes sense to me, seeing how a single disease like chestnut blight or phylloxera can be very devastating to a population on a whole continent, it could be both as bad but also less noticeable in a natural park.