The referer header is also controlled by the client. Anything controlled by the client should be considered tainted.
In a CSRF scenario, the client itself can presumably be "trusted" - A client behaving maliciously can only hurt itself, not any one else. The important thing is to not trust anything that may have been provided by a third party.