Not as fast. Chacha20 uses 32-bit additions which are fast in software but expensive and slow in hardware. In addition protecting Chacha20 from power analysis attacks is more difficult compared to AES.
> just generic SIMD
Constant-time AES with SSE2 is actually faster than the naive variable-time AES. See https://www.bearssl.org/constanttime.html#aes
In addition Chacha20 is not nearly as fast as AES when using the AVX-512 Vector AES instructions.
> but also Adiantum
Which uses AES (once per sector).