If you want to allow for high scores, you could let people collect score over unbroken chains of victories.
Yep I think this is coming together. I won't ditch the idea of having a score, but I will ditch the idea of competing against your opponent based on score. You won't even see your opponent's score, only yours. You compete for survival. And the accumulated score will be a sort of trophy :)