Then keep them detached with the Matrix ID <-> Custom ID <-> Phone, in DB relationship parlance.