This sounds like what Microsoft research did with SEAL to produce CryptoNets
https://www.microsoft.com/en-us/research/publication/crypton...
https://www.microsoft.com/en-us/research/publication/crypton...
[1]: https://www.cs.utexas.edu/~roshan/CHET.pdf [2]: https://arxiv.org/pdf/1812.10659.pdf