What are you storing in your contacts database that's so huge that it can't be loaded over the network every time?
Are you hand-writing notes and storing the notes as images?
I use Apple/Google's built-in notes fields on the default addressbooks and it works just fine. I can't imagine having huge write-ups on individual contacts unless it was for some business purpose. In that case, I'd move to a dedicated note-taking application anyway.