Does brew compile from source? That is one of the most frustrating aspects of macports, for example I was trying to install a simple musical ear training tool the other day and it went to compile GCC. Argh!
Once you get all that, it goes faster, but the first time you go to install something simple you're basically going to walk away for a couple hours while it sets up its own little world.