Signal's developers don't want to store contact lists on their servers because of their concerns about privacy. They also worry about ease of use for non-technical users.
Their solution is to use the end user's address book as a contact list with users' phone numbers acting as identifiers. The identifiers are mapped to endpoints with some method that is supposed to protect the end user's privacy.
The app could encrypt a list of usernames and store it on a server, but that would affect usability for non-technical users.