One of the biggest reasons why I like Google Photos is all the processing that it does on photos, especially some of the features that you have mentioned (object and face detection, location clustering, image filters, ...).
Now to process these photos, you would need to read them and since you are end-to-end encrypted it's up to the clients to do this processing. Would some of these features even run on mobile devices(or for that matter javascript on the web)?, since google uses AI heavily for these tasks. You upload a picture on google, and instantly you get all the processing done on your pictures and they are available for you to browse and search. Google uses custom built AI processors and massive GPUs to get that computation done quickly. To replicate that in javascript on the web and mobile devices is going to be hard since there are few libraries which support it and the mobile devices really cannot compete with the computing power available in the cloud.
I really love the privacy oriented aspect of this service though and I would really like to share one less thing with google. I've always been concerned about being blocked out of my google accounts and losing my photos.