It doesn't make sense only because you put it in terms of some completely abstract and made-up notion of "scrolling the window". Put another way:
One finger drags the cursor up and down
Two finger drags the underlying page up and down
Four finger drags the underlying space left, right, up or down
Perfectly consistent, no? I won't argue that forcing people to relearn scrolling is an uphill battle. But I'm a Dvorak typist.