If I were to drop CSRF tokens (which work reliably) I would go directly to Origin verification. No Referer. Something in the middle is worst of both worlds.
EDIT: when Firefox _does_ start sending the Origin header, we may drop the Referer - we'll see.