The seminal definition (must start with a letter) was from RFC 952. Then it was lifted in RFC 1123: “One aspect of host name syntax is hereby changed: the restriction on the first character is relaxed to allow either a letter or a digit. […] If a dotted-decimal number can be entered without such identifying delimiters, then a full syntactic check must be made, because a segment of a host domain name is now allowed to begin with a digit and could legally be entirely numeric (see Section 6.1.2.4 [sic]). However, a valid host name can never have the dotted-decimal form #.#.#.#, since at least the highest-level component label will be alphabetic.”
Note that section 6.1.2.4 is completely irrelevant, it looks like a referencing mistake. The later RFC 3696 then calls the rule the LDH rule and says: “There is an additional rule that essentially requires that top-level domain names not be all-numeric.”
But it never says where such a rule comes from.
RFC 2181 references section 6.1.3.5 of RFC 1123 (maybe that’s an old 6.1.2.4) but it doesn’t state that rule either.