> it's likely that the extension itself is harmless enough
That seems unlikely given Adobe's history of truly awful security flaws. It wasn't that long ago when they thought that it would be a good idea for their add-on to pre-render PDFs in RAM silently in the background, including executing any embedded code without any sandboxing. Combined with browsers' prefetching of urls in a page (so that it would load quicker in case you clicked it), this caused a number of rootkit and other malware infections - from links that people didn't even click in search results and URLs served up in advertising or in comments/forum posts.
The only permission needed by a PDF viewer should be 'display PDF document content'. It shouldn't need to read or change other data, manage downloads, or communicate with anything to display an e-book or document. If it does, it's probably not harmless.