You cannot both a) forge the Referer header and b) trick the victim's browser into submitting a request - at the same time.
It's true that the REFERER header isn't always included in requests, BUT when it is included, you can consider the request cookies you receive plus the REFERER header accurate when taken together, barring a browser vulnerability. If there is no browser vulnerability, the attacker doesn't get to mess with the REFERER header on the victim's browser.
Note: It is NOT safe to trust an empty REFERER header.
Never. Trust. User. Input.
I'm not saying referrer checking is useless, but once again, you never trust user input. Ever. If you follow that rule you're decreasing attack vectors to essentially social engineering, brute force, and complex technological exploits using buffer overflows and the like (which I guess is still kind of a side effect of trusting user input).
You could indeed possibly forge a request after harvesting session cookies - but that isn't CSRF. It's either XSS or traffic snooping.
One way to cut down the (time) window of opportunity for an attack is to have the server generate new session tokens, and store them in the cookie, on each request, but require a valid token in order for a new token to be generated. If you have an XSS vulnerability, this is not an adequate solution, as it requires the victim to submit another request between the time the attacker steals the cookie and the time the attacker submit the cookie to the server.
So basically, use SSL and be really careful about XSS. You can trust the REFERER header and session cookies together if and only if they are both present. "Never trust user input that you have to escape or that will be rendered to the browser" is very reasonable though.