This may be a reason why babies stick everything in their mouths.
This is the basis of oral immunotherapy, and if you ate latex daily it could possibly desensitize you. However, the immune system is insanely complicated and not fully understood. There are a lot of gotchas here. It may actually be possible to desensitize with skin exposure with careful control of the dose, as there are some 'skin patch" treatments that work for some people although generally not nearly as well as the oral route. Not all allergies are the same, and may not be treatable by exposure in some people. The immunity obtained by immunotherapy may not be the same as natural immunity, it may disappear over time, and the treatment itself can have hard to detect but severe chronic side effects like eosinophilic esophagitis. So don't DIY!
Interestingly I have heard that mango skin contains the same irritant chemical as poison oak. I wonder if eating mango skin would help desensitize people to poison oak. I once ate a very small amount by accident and had a weird feeling in my throat and a bad taste in my mouth for ten minutes afterward, so it sounds pretty unpleasant to me.
Latex may be an exception depending on the mechanism of action, but almost all organic compounds that can be metabolized by your body can be adapted to.
Allergies occur because the immune system incorrectly associates a substance with a bad reaction, and so attacks it as it would a pathogen. The problem is that the allergen isn't a pathogen, and so the immune system can't actually kill it.
There's a treatment for pet/pollen/etc. allergies that works by injecting a very small quantity of the allergenic substance every week, slowly building up tolerance. The body learns that the small dose didn't cause problems, and slowly gets accustomed to higher and higher doses. If the doctor sets the dosage too high, the body has an allergic reaction and then that allergic reaction reinforces the immune system's determination that the substance is dangerous. The treatment response to this is to drastically reduce the dosage and try again.
If the sibling comment's assertation that oral exposure was desensitizing was correct, that wouldn't explain why some people develop food allergies later in life. (As one anecdotal example, my Wilderness First Responder instructor was slowly getting more and more allergic to mangos.)