1. It is so modular that you cannot really optimize it using something like Google Closure (http://code.google.com/closure/ ). If you do, it breaks it.
2. The modularity means that you're likely to need a few roundtrips to the server to get everything. Invariably these block the loading of the page and you get a hang, sometimes for a few seconds. This is not good for user experience.
Of course MathJax solves a very important problem, but it's a problem browsers have introduced. None of the support MathML in a sane way: Firefox requires content served as fully valid XHTML (good luck with that if you're using a CMS) and Webkit only got MathML support recently in the nightly releases. If browsers actually implement MathML in a way developers can use without losing hair, then MathJax is no longer needed. Till then, I love MathJax despite its warts.