It seems to me like you’re easily hand waving away a hard problem in a different part of the stack you’re less familiar with.
It seems to me like you’re easily hand waving away a hard problem in a different part of the stack you’re less familiar with.
Again, the novelty is in getting cross attention to work correctly despite the fact that you’re stitching together arbitrary caches together. It’s akin to taking snippets of compressed portions of random compressed files and reconstructing a new correct plain text. That’s obviously not possible but clearly this has been accomplished with the KV cache for arbitrary models (ie not trained for it) despite the KV cache working like decompression where all the preceding bytes have to be computed correctly for the subsequent token to be correct.
Meanwhile the AI engineers are doing the exact opposite. Bragging about the hard parts and rolling their eyes at the easy parts.