I have been working on this problem for the last year. As it turns out this problem is especially prominent when doing object detection in the real world on a drone platform. Besides a large number of angles/contexts being able to move in 3D adds a large number of scales also, and there is hardly any open training data besides the VisDrone dataset[0] which doesn't even address the scale issue.

It is certainly an interesting problem though. I can't talk much about my work but if anyone wants to collaborate on something open source addressing the core problem check my profile.

[0]: http://www.aiskyeye.com/upfile/Vision_Meets_Drones_A_Challen...