Thanks for sharing, that was a great read. I've always really been fascinated by the aquatic ape hypothesis (https://en.wikipedia.org/wiki/Aquatic_ape_hypothesis), even if it's not widely accepted as fact.
218 karma · joined December 4, 2022
The prospect of low cost swarms of AI enabled drones terrifies me. If you've watched any drone combat footage from Ukraine you know it's just around the corner, and this is another step in that direction.
Cutting energy costs by an order of magnitude could be enough motivation, but you'll still need to demonstrate competitive model performance.
2. I find the QKV nomenclature unintuitive too. Cross attention explains it a bit, where the Q and K come from different places. For self attention they are the same, but the terminology stuck.