Why is Transformer training more parallel?

The bottleneck

Recurrent states depend on preceding states, limiting parallelism over positions. The 2017 Transformer connects positions through attention and expresses each layer as matrix operations. This removes within-layer recurrence, not dependencies between layers.

The mechanism

During training the target sequence is known. Shifted ground-truth tokens are inputs and a causal mask hides future tokens. QKᵀ scores positions, and softmax weights V. During decoding future tokens are unknown: ordinary autoregressive generation still produces them sequentially. KV caching avoids recomputing past keys and values.

Attention(Q,K,V) = softmax(QKᵀ / √dₖ + mask)V

Examples and evidence

Teaching example: training [A,B,C]→[B,C,D] can compute losses at all three positions in one forward pass. Generation must obtain B before using B as the next input. Actual parallelism depends on batch size, sequence length, hardware and implementation.

New constraints

Dense attention scores grow quadratically with sequence length. More parallelism does not mean less total computation; long contexts introduce memory, IO, positional representation and data-coverage constraints.