I recently saw a paper (which I reposted here on HN a while back) that perhaps could help with your problem. The paper proposes an approach that sort of induces models to learn representations of objects that are good at predicting "the most agreed-upon" rotations of the inputs. I'm not explaining it well. Anyway, a model in the paper achieved SOTA on a change-of-viewpoint dataset with a really tiny number of parameters. Might be worth a look: https://arxiv.org/abs/1911.00792