Homebrew will use libraries that are already present on your system, instead of keeping a separate copy of everything.

The downside may be decreased stability, but the reduced compilation times make up for that, to me (and haven't had a problem yet).