With Chacha20 the worst that can happen is plaintext recovery, which is a lot less bad than full key recovery.