It is said that Solon, the most revered law-giver of Athens, made cutting an olive tree punishable by death. I've also heard that because of their slow growth, planting an olive tree is something you do not for yourself but for your children.

EDIT: one more bit of historical perspective. Myth says that Athena and Poseidon had a contest to see who would be the patron of Athens. Poseidon gave the gift of a saltwater spring (to which I guess the Athenians said, "uhh, thanks, I guess."). Athena gave the gift of the olive tree. Obviously Athens is named after her. So that's how much they valued those trees.