There are plenty of competitive weight lifting sports. CrossFit, Olympic lifting, powerlifting...
That's more like the weight lifting you're comparing it to - the competition involves artificially choosing the other person as a benchmark, and I can work against whatever internal benchmark I want.
This is very different than a game of one-on-one, where it's his job to change how I interact with the ball (which he would, to the point of meaninglessness) and vice-versa (which I would fail at to the point of triviality).
It's not strange (or a personal failing) to find that less interesting.