Also, now I feel very old. "Little black book" was an ubiquitous term when I grew up, and so using the term would just be a synonym to "address book" that nobody would think twice about.
[1] E.g. The House and Home: A Practical Book - Volume 1 (1896): "And there is a little black book with red lettering seen on every writing-table and carriage-cushion wherein puzzled mater-familias finds her bearings annually among her cherished acquaintances, many of whom the little black book alone keeps in her recollection!"
[2] Guess you got back (Guess you got) To my name (To my name) In your little black book