FLOP counts are the wrong unit for planning an inference deployment, and attention is the clearest example: a small share of the arithmetic, a large share of the time.
The measurement
Five frontier MoE models, four accelerators, one roofline per layer. We report arithmetic intensity, achieved bandwidth and achieved FLOP/s side by side, so each layer lands somewhere legible on the roof.
What it implies
If attention is memory-bound at your sequence length, buying arithmetic throughput buys you nothing. The decision that matters is layout and KV movement, and it is invisible in a FLOP budget.