JPEG exploit: https://www.f-secure.com/v-descs/ms04-028.shtml
http://www.joelonsoftware.com/items/2008/02/19.html
"These are binary formats, so loading a record is usually a matter of just copying (blitting) a range of bytes from disk to memory, where you end up with a C data structure you can use. There’s no lexing or parsing involved in loading a file. Lexing and parsing are orders of magnitude slower than blitting."
PSD is much the same - it's why interop with PSDs is such a challenge.
JPEG has to be parsed and decompressed - another source of problems, but not remotely in the same category.