Quarantine had me running around in high school reading Quantum Physics books and mostly failing to understand them. Good book.
Egan's two most recent books, Incandesence and The Clockwork Rocket, are great if you're a math and physics guy.
If you're a programmer, Permutation City would be my recommendation. If you're more mathematically inclined, Diaspora is a good choice. Just so you know what kind of book you'd be in for, the first chapter of Diaspora has a proof outline of the Gauss-Bonnet theorem. His short story The Planck Dive gives a good flavor of Diaspora: http://gregegan.customer.netspace.net.au/PLANCK/Complete/Pla...