How should a fixed compute budget split parameters and tokens?

The bottleneck

The 2020 scaling-laws work models loss against parameters, data and compute. Chinchilla in 2022 found many large models undertrained relative to their size; spending a fixed budget on more parameters can be worse than training on more tokens.

The mechanism

Approximate dense decoder training compute as C≈6ND, where N is parameters, D is training tokens, and 6 captures dominant forward/backward work. Evaluate loss for different N,D at the same C and fit an allocation. Chinchilla supports scaling both within its experimental regime, not an eternal token-to-parameter ratio.

C ≈ 6 × N × D  [FLOPs]

Examples and evidence

Teaching example: 10B parameters × 200B tokens is approximately 1.2×10²² FLOPs; 20B × 100B uses the same approximation. Equal approximate compute does not imply equal quality. This omits sequence-dependent attention, sparsity and hardware utilization.

New constraints

Training-loss optimality is not lifecycle-cost optimality. A high-volume service may train a smaller model longer to reduce per-call inference cost. Available high-quality, nonredundant data is another constraint.