Shouldn't the memory needed scale quadratically with sequence length rather than the linear scaling they have in their equations?
> GPT-NeoX achieves 150 TFLOP/s/A100 with normal attention and 180 FLOP/s/A100 with Flash Attention.
This advice implies they are using flash attention.