Usually this is done by sending an nonce that must be sent back with the request to delete the account. The attacking site won't have the nonce, so it cannot delete the account.
Currently, you can also get a similar protection by requiring headers such as "Content-Type: application/json", since other sites will not be able to make that request.
Referer does work in some cases, but it is fairly common for users to disable the Referer, so relying on it for site functionality is not ideal.