>The Xbox gpu and apu are 'off-the-shelf' in the sense that they were re-used in nvidia retail chips.
Not really[1]. It was based on the GeFroce 3 architecture (NV20) but the actual chip had been quite different from the ones you could get on a retail card (NV2A). Even if it had been a straight off-the-shelf device it does not mean you know how it works. Look at the open-source drivers for the off-the-shelf video cards to see what is possible through reverse-engineering and what is not.
I cannot speak for other consoles of that generation, as I have not touched the GC but PS2 devkits came with nice printed manuals, describing every register and every command for almost every device (if memory serves there was not printed IOP manual but it was pretty much the PS1 chipset so its documentation had been obtainable too).
1. https://en.wikipedia.org/wiki/GeForce_3_series