Any chance of not requiring access to the whole contact list? I think there are pickers that allow selecting one contact - or better yet, one could allow adding contacts inside Signal.
Apple offer a similar tool when leaving iOS https://selfsolve.apple.com/deregister-imessage