Yes, I have spent a lot of time out in nature and parks, and I generally am aware of not leaving a trace or litter. However, with something like a banana peel, I'd rather it end up as tree food the in a landfill, so I usually toss them in bushes.
Apple cores I think might not be too bad to toss, ants can eat that up in a day or two.
I pack out the peels of any tropical fruit.