If you really love self-hosted, SugarCRM is the main open source CRM out there.
Sugar has its own problems: https://news.ycombinator.com/item?id=14181662
Agreed that most apps in this space are huge and focused on business and sales. I am more interested in software that helps cultivate personal, offline relationships -- as opposed to ScaleMail and such.
For me it would probably be the app proactively reminding me "hey, you marked this person as important but haven't contacted them for a month". Even if this can become a bit annoying it still keeps you in the loop and puts a thought at the back of your head.
<insert it's something meme here>
Another one would probably be the app automatically stalking the person and informing you if they're about to participate in an event where a keyword / tag matches your interests. (That sounds pretty tough to do however.)