When I was making a simple EPUB app for my own use [1], I found it surprising that many publishers don't follow IDPF standards. Most do, but there are a significant amount that mix and match all sorts of rules.
What this leads to is basically fixing random, one off issues depending on publisher and book. I would definitely suggest not writing your own reader and instead looking at something like Readium (and contribute if you have time!).
[1] http://mademers.com/two-more-indesign-cs5-export-to-epub-bug...