When I did it, I had my DPhil grant/post-doc salary and it felt like side money rather than being my 'job' and I was partly doing for the experience, for my CV (good to be able to put 'Lecturer at Oxford' on it!) and partly because I enjoyed it. I felt quite lucky to be offered tutoring jobs at Oxford as they were much sought-after by DPhil students.
I think it's also worth pointing out that not all Oxford colleges are 'rich' and many have buildings and grounds that are expensive to maintain. I don't think most of them are better placed to pay more than other UK universities.
There's an argument to be made that people teaching at universities should be paid more, but it does feel unfair to single out Oxford.