(1) Code spends a huge amount of time in a non-parsable state. I need to be able to save code even when it's not currently parsable into a valid AST of the language I am writing. Sometimes I even need to check in this code to my source control. You cannot have the primary storage of your code be valid, parsed syntax trees for this reason, so you need to invent a brand new format that is different from both text and the AST being modeled. This format needs to be optimized for programmer workflow, clarity, and communication; not optimized for the machine.
(2) You need to re-invent how source control works for your representation if you're not using text. This actually has a lot of potential -- syntax-aware merging done well would be a godsend. But this is a huge undertaking to do well, and problem 1 rears its head strongly here as well.