I'm from Jamaica, and grew up eating lots of lychees as well. Never had any problems. Only ate them ripe. I haven't had fresh lychees in years though, mostly now buying the canned ones in syrup from China.

The owners of lychee trees, however, had tons of issues. It's such a loved fruit that if you had such a tree, it was pretty much guaranteed that praedial thieves would strike. It got so bad that I remember one nurse in the community cutting down her tree when the sight of strange men roaming her yard became too much.