There are easily found examples of malicious content being injected into HTML -- malvertisements for example. I can only imagine what might get injected into a PDF which can run javascript [1]. PDF readers aren't exactly known for their security.
Frankly, I'd much rather be able to talk to you about something downloaded from your site and get you to fix it instead of allowing a third party to infect me and point fingers at you.