There are various ways to deal with this in ML (not really an expert in ML), but AFAIU in most cases you get candidate transforms via model-based methods, and then make the decision using ML-based methods trained to allow for limited transformations (convolution nets are good at that).
What research is out there on general Affine invariant vision algorithms and techniques?
For context, my practical experience ended in the mid 2000's when "jittering" was a bit of thing along with some occasional closed form estimators based on basic linearization approximations.
SIFT and its multiple descendants, yes, and also things like Random Ferns, google "Boosted Random Ferns for Object Detection" (it's damn difficult to get a clean link to pdf from google...)
Okcupid talked about this in an article about image hashing [0] and they have a nice quote:
"The end-to-end approach of training a Convolutional Net to embed images such that the positive / negative case distances separate nicely is probably the cool new hypebeast way forward. This would have the added advantage of being able to handle whatever transformations you can think of at training time."
[0]https://tech.okcupid.com/evaluating-perceptual-image-hashes-...