Instead of having your backend download and retransmit the file, you could return a redirect to it instead? Plus maybe lower the poll rate from a few times per minute to once an hour or so to avoid the need for caching to save battery.
But that would still expose your authentication credentials to the backend, so a malicious backend could MITM you.
Reducing the polling is definitely an option, I want to make that configurable very soon.
Btw, the cached data in the backend is encrypted with a token that is only transmitted from the device to the backend during the API request of the device. It's not end-to-end, but it's a step.