Important questions

2017–2026: training, inference and systems. Bottlenecks, breakthroughs and new constraints.

Content revision 4 · 2026-10-24

Train · Training

Separate parallel training positions from autoregressive generation.

Understand scaling laws and Chinchilla through a compute budget.

Distinguish total parameters, active parameters, routing and communication.

Move from text prediction to demonstrations and preferences.

Understand implicit rewards and offline preference pairs.

Distinguish R1-Zero from the multistage R1 pipeline.

Turn cache capacity into a checkable MiB calculation.

Shared KV heads and latent compression are distinct designs.

Acceptance, rejection and correction all matter.

Test window size, positions and retrieval separately.

Allocate budget across serial reasoning, candidates and verification.

Account for weights, activations and KV precision separately.

Remove redundancy in parameters, gradients and optimizer states.

Choose device layouts by partition and communication path.

Distinguish FLOPs, memory IO and intermediate storage.

Logically consecutive tokens need not occupy contiguous memory.

Schedule at iteration boundaries rather than whole-batch completion.

Compare independent scaling with state-transfer cost.