My suspicion is that Intel's manuals used hex because octal was kind of obsolete by that point.
i agree about the registers and rst instructions
i don't see how octal could be obsolete; computers still use binary, and octal is just as useful for doing binary arithmetic as it ever was. as you point out, it's considerably more useful for instruction sets with 3-bit fields. moreover, the great advantage of hexadecimal was that bcd was easy to read, but that stopped mattering around the time the 8008 was introduced
i'm interested to hear if your experience differs, but i think it's a lot easier to do mental arithmetic in octal than in hex; it's a lot easier to remember 5 + 6 = 13, or figure it out if you forget, than d + c = 19, and both the addition and multiplication tables are one fourth the size
as far as i can tell, octal just went out of style. if anything, hexadecimal became obsolete, because its killer advantage went away, while its drawbacks remained
arm leaned into the hexadecimal wave and designed their instruction set to be easily readable in hex. we probably don't have to worry that a possible wide adoption of risc-v will initiate a transition to base 32, though; its instruction formats, while much simpler than thumb, resists any such easy numerical readability
btw, i don't remember if i ever told you this, but thank you for xfractint
i had a heathkit h8 which had an unusual approach to this problem
it displayed memory addresses on its front panel in octal, from 000000 up to 377377
A 9-bit byte is subdivided into three 3-bit nibbles/digits (2^3=base-8) [000, 777].
An 8-bit byte (octet) is evenly subdivided into two 4-bit nibbles/digits (2^4=base-16) [00, FF].
In documentation--given that the majority of computers use 8-bit bytes--the base we choose to represent 3-bit fields is a separate topic, and could be called style.
as a side note, i have a quibble with your definition of 'byte'. lots of computers have been word-oriented, with words being generally the smallest addressable unit
word sizes in word-addressed computers have included 12 bits (pdp-8), 16 bits (nova and tandem), 18 bits (pdp-7 and ga144), 20 bits (mup21), 32 bits (zuse z4, chifir, and maybe some ti dsps, not sure), 36 bits (pdp-10, ibm 704, ibm 709), 60 bits (cdc 6600), 64 bits (cray unless you count the parity bits), and so on
it was not normal practice to describe any of these words as a 'byte'; more commonly, you would divide the word into (possibly variable-size) 'bytes', each representing one character, even though you couldn't load those bytes individually from memory