But why bother putting the user id in the AD instead of part of the authenticated encrypted payload?
That guarantee is not provided with unauthenticated stream ciphers. For example, some stream ciphers work by essentially using a deterministic but unpredictable PRNG seeded with the key and IV to generate a bitstream, and then XOR the plaintext with that bitstream to generate the ciphertext.
With such a stream cipher, If Eve knows that the data format is, e.g., the an 8-byte unsigned integer user ID followed by the rest of the payload, Eve can take the first 8 bytes of the ciphertext and XOR it with Bob's user ID (public information) and her own user ID to corrupt the message in such a way that the ID in the resulting cleartext would contain her user ID instead of Bob's, and thus pass the validation that it seems like you are proposing.
Let C[] be the cipher text, K[] be the key stream, B be Bob's ID, and E be Eve's ID:
C[:8] = B ^ K[:8]
C[:8] ^ B = K[:8]
C' = C[:8] ^ B ^ E ++ C[8:]
C' would decrypt, validate as "belonging" to Eve, and contain Bob's data.
If you are using an AEAD cipher mode, then you always have AD, but sometimes that AD might be the empty string. In that case, the advantage to using contextual AD as opposed to using the empty string as AD and then doing additional verification on the decrypted object is that it prevents some kinds of timing attacks, because cryptographic libraries will often implement AEAD constructions to fail in constant time, where as your scheme will take longer if post-decryption validation of contextual data encoded in the plaintext fails compared to if decryption fails.
OK fair.
> and I would consider whatever "secret key" you pass into HMAC along with the ciphertext to be the "associated data"
The secret key isn't associated data. You take your base HKDF key and expand new crypto for an authenticated cipher from the offset as info (+ maybe other parameters like file name). That key is then used to decrypt. If you squint I guess you could call that AD but it's functionally a very different role.
> because cryptographic libraries will often implement AEAD constructions to fail in constant time, where as your scheme will take longer if post-decryption validation of contextual data encoded in the plaintext fails compared to if decryption fails.
I think you've misunderstood what I said. As I repeat above, the AEAD key is derived from the offset. There's no post-decryption validation of contextual data because the plaintext is empty. HKDF derivation is constant time and authenticated decryption is constant time. Once decrypted you have a valid block at that location. There's nothing extra left to validate (or perhaps the decrypted contents, but that's irrelevant for cryptographic purposes).
My broader point is that I have yet to encounter a use-case for a non-empty AD string.
As you said, same effect: the scheme you have described is not an alternative to AEAD. It is an example of AEAD. You're still using the offset as associated data, you just happen to be composing your AEAD scheme out of another AEAD scheme into which you pass an empty string as associated data.
Other than differences in the limit to the number of messages you can encrypt before nonce exhaustion or the number of bits of secrecy or authentication strength provided, the external interface and use case of your system perfectly matches that of AES-GCM or other popular AEAD constructions.
Ahh, but that’s not a trivial part of the design and why this is strictly better than using a single AES-GCM key with AD. And also it’s more generic across whatever type of key you choose to derive.