LLM technical evolution

Training

Specific disclosures appear below; other implementation details remain unknown.

reasoning_rl-o1 · The current card discloses reasoning RL (Sol by explicit reference); the optimizer is unspecified.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2 Model Data and Training · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

thinking-internal · Internal reasoning precedes the answer; prompted CoT and training are not equated.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2 Model Data and Training · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

reasoning_rl-o1 · The current card discloses reasoning RL (Sol by explicit reference); the optimizer is unspecified.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2 Model Data and Training · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

thinking-internal · Internal reasoning precedes the answer; prompted CoT and training are not equated.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2 Model Data and Training · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Constitution-guided post-training is described; the exact recipe is not disclosed.

post-opus55 · Substantial constitution-guided post-training; no undisclosed optimizer is assumed.

This does not establish every step of the historical CAI recipe.

Claude Opus 5.5: training methodology · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

thinking-adaptive · Adaptive thinking is always on, with depth controlled by effort rather than the old manual budget.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

New model / Breaking changes · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

moe-sparse · Explicitly a sparse MoE Transformer with native multimodal support.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Architecture · Relevant text reviewed · Checked 2026-10-23

reasoning_rl-o1 · RL uses multi-step reasoning, problem-solving and theorem data.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Training Dataset · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

Specific disclosures appear below; other implementation details remain unknown.

system-argon · TPU, JAX and ML Pathways are explicitly used for training.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Hardware / Software · Relevant text reviewed · Checked 2026-10-23

Training

Specific disclosures appear below; other implementation details remain unknown.

post-glm53 · Uses the 5.2 base with improvements from post-training.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Introduction: GLM-5.3 · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

hybrid-glm53flash · The new base combines sparse / linear attention and mHC.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Introduction: GLM-5.3-Flash · Relevant text reviewed · Checked 2026-10-23

repr-mhc · The official description explicitly names mHC.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Introduction: GLM-5.3-Flash · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

hybrid-glm53flash · The same structure combines recurrent state and sparse access during inference.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Introduction: GLM-5.3-Flash · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

Specific disclosures appear below; other implementation details remain unknown.

context-csa2 · Full, Reindex and Reuse control sharing and index recomputation.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2.3–2.3.1 · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

Specific disclosures appear below; other implementation details remain unknown.

context-csa2 · Shared main KV and index keys reduce storage; index reuse avoids repeated computation.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2.3.1 · Relevant text reviewed · Checked 2026-10-23

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

hybrid-k3 · KDA and Gated MLA mix at 3:1, with an additional final global layer.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2.1 · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

hybrid-k3 · Recurrent state updates complement global access; this is not a purely linear model.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2.1 · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

hybrid-q38 · Specifies Gated DeltaNet, Gated Attention, MoE and multi-step MTP.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Model Overview · Relevant text reviewed · Checked 2026-10-23

Inference

Specific disclosures appear below; other implementation details remain unknown.

thinking-adaptive · reasoning_effort adjusts reasoning and thinking history can be preserved; no claim of identical internals to Claude.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Highlights / API Usage · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

Specific disclosures appear below; other implementation details remain unknown.

context-msa · Block-selected sparse attention.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

MSA · Relevant text reviewed · Checked 2026-10-23

Training & serving systems

Specific disclosures appear below; other implementation details remain unknown.

context-msa · A KV-outer kernel gathers matching queries to reduce repeated reads.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

MSA · Relevant text reviewed · Checked 2026-10-23

Training

Specific disclosures appear below; other implementation details remain unknown.

post-wonder · The official post explicitly describes scaling pretraining and reinforcement learning; the optimizer is unspecified.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Scaling Toward Personal Superintelligence · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

post-wonder · The official post explicitly describes scaling pretraining and reinforcement learning; the optimizer is unspecified.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Scaling Toward Personal Superintelligence · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

agent-k2 · RL spans tool environments with tests, judges and static checks; this is not adoption of Kimi’s recipe.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Reinforcement learning at scale · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

Specific disclosures appear below; other implementation details remain unknown.

system-ml4 · Rollout actors and training execute asynchronously with staleness control.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

Reinforcement learning at scale · Relevant text reviewed · Checked 2026-10-23

Training

Specific disclosures appear below; other implementation details remain unknown.

train-ernie5 · Unified next-group-of-tokens training with modality-agnostic expert routing.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

§2–2.1 · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

Specific disclosures appear below; other implementation details remain unknown.

moe-sparse · Officially identified as sparse MoE; the exact routing strategy is not established.

Supports the described mechanism or application, not every historical implementation detail or weight lineage.

核心Capability与技术参数 / Architecture · Relevant text reviewed · Checked 2026-10-23

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

Training

The reviewed material does not establish the implementation; no link does not imply non-use.

Inference

The reviewed material does not establish the implementation; no link does not imply non-use.

Training & serving systems

The reviewed material does not establish the implementation; no link does not imply non-use.

2017-06-12 · Transformer · Method

Attention architecture · Transformer

Bottleneck

Recurrence limits parallel training and lengthens information paths.

Mechanism

Replace recurrence with attention to process training positions in parallel.

Evidence and conditions

The 2017 paper introduces an attention-only sequence-transduction architecture. Parallel training does not remove autoregressive generation dependencies.

New constraints

Quadratic attention and serial autoregressive generation remain.

Primary sources and supported content

Primary source: method and results

• Attention Is All You Need · 2017-06-12 · Checked 2026-10-21

Relations

• Alternative · BERT

  Bidirectional encoding and causal generation use different attention visibility and objectives.

  • BERT · 2018-10-11 · Checked 2026-10-22
  • Attention Is All You Need · 2017-06-12 · Checked 2026-10-21

2020-01-23 · Scaling laws · Method

Scaling laws · Scaling Laws

Bottleneck

Compute allocation across parameters and data lacks an empirical basis.

Mechanism

Fit loss scaling with model size, data and compute, and study fixed-budget allocation.

Evidence and conditions

The 2020 work establishes measurable scaling relationships. Later training experiments revisit its empirical regime and allocation conclusions.

New constraints

Fitted laws depend on data and experimental ranges; extrapolation is limited.

Primary sources and supported content

Primary source: method and results

• Scaling Laws for Neural Language Models · 2020-01-23 · Checked 2026-10-21

Relations

• Improvement · Chinchilla

  Re-estimate parameter/data allocation under fixed compute.

  • Training Compute-Optimal Large Language Models · 2022-03-29 · Checked 2026-10-21

2020-05-28 · GPT-3 · Model practice

Scaled pretraining · OpenAI · GPT-3

Bottleneck

Fine-tuning each task requires labeled data and model updates.

Mechanism

Scale autoregressive language modeling and specify tasks through in-context instructions and examples without evaluation-time gradient updates.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Prompt sensitivity, contamination and deployment cost remain.

Primary sources and supported content

Abstract: method and scope

• Language Models are Few-Shot Learners · 2020-05-28 · Checked 2026-10-22

Relations

2022-03-29 · Chinchilla · Method

Compute allocation · Chinchilla

Bottleneck

Some large models have too many parameters and too few training tokens for fixed compute.

Mechanism

Jointly choose parameters and training tokens within a fixed compute budget.

Evidence and conditions

The 2022 study compares parameter/data configurations and identifies undertraining relative to model size; it does not prescribe a permanent ratio.

New constraints

Training optimality excludes some serving costs; more high-quality tokens may be unavailable.

C ≈ 6ND
N: parameters · D: training tokens

Primary sources and supported content

Primary source: method and results

• Training Compute-Optimal Large Language Models · 2022-03-29 · Checked 2026-10-21

Relations

• Improvement · Scaling laws

  Re-estimate parameter/data allocation under fixed compute.

  • Training Compute-Optimal Large Language Models · 2022-03-29 · Checked 2026-10-21
• Alternative · LLaMA

  Optimal fixed training compute and optimal lifecycle inference cost are different objectives.

  • Training Compute-Optimal Large Language Models · 2022-03-29 · Checked 2026-10-21
  • LLaMA · 2023-02-27 · Checked 2026-10-22

2023-02-27 · LLaMA · Model practice

More training data · Meta · LLaMA

Bottleneck

Optimizing training compute alone can leave an expensive model to serve.

Mechanism

Train smaller models on more tokens, considering training cost alongside repeated inference cost.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

More data and training time are needed; this is not a unique fixed-training-compute optimum.

Primary sources and supported content

Introduction / training efficiency

• LLaMA · 2023-02-27 · Checked 2026-10-22

Relations

• Alternative · Chinchilla

  Optimal fixed training compute and optimal lifecycle inference cost are different objectives.

  • Training Compute-Optimal Large Language Models · 2022-03-29 · Checked 2026-10-21
  • LLaMA · 2023-02-27 · Checked 2026-10-22

2017-01-23 · Sparsely-gated MoE · Method

Sparse expert routing · Sparsely-gated MoE

Bottleneck

Capacity growth usually increases per-token compute.

Mechanism

A gate activates a small expert subset, partly decoupling capacity from per-token compute.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Imbalanced routing, expert capacity and communication become constraints.

Primary sources and supported content

Abstract: method and scope

• Sparsely-Gated Mixture-of-Experts · 2017-01-23 · Checked 2026-10-22

Relations

• Improvement · Switch Transformer

  Simplify per-token expert choice and routing complexity.

  • Switch Transformers · 2021-01-11 · Checked 2026-10-21

2020-06-30 · GShard · Method

Expert sharding · GShard

Bottleneck

Sparse models are difficult to shard manually.

Mechanism

Sharding annotations and automatic parallel compilation scale conditional MoE computation.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Expert capacity and all-to-all communication still limit scaling.

Primary sources and supported content

Abstract: method and scope

• GShard · 2020-06-30 · Checked 2026-10-22

Relations

2021-01-11 · Switch Transformer · Method

Single-expert routing · Switch Transformer

Bottleneck

Multi-expert routing adds communication and instability.

Mechanism

Use top-1 routing to activate one feed-forward expert per token and scale conditional capacity.

Evidence and conditions

Switch studies simplified sparse routing, stability and load balance. Total parameters and per-token activation must be counted separately.

New constraints

Single-expert routing still needs capacity management, balancing and numerical care.

Primary sources and supported content

Primary source: method and results

• Switch Transformers · 2021-01-11 · Checked 2026-10-21

Relations

• Improvement · Sparsely-gated MoE

  Simplify per-token expert choice and routing complexity.

  • Switch Transformers · 2021-01-11 · Checked 2026-10-21

2024-01-08 · Mixtral 8×7B report · Model practice

Sparse expert models · Mistral · Mixtral

Bottleneck

More capacity is needed within active-compute limits.

Mechanism

Each layer selects two feed-forward experts per token while sharing attention; the report gives model-specific quality and speed conditions.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Total weight memory does not fall in proportion to active parameters.

Primary sources and supported content

Abstract: method and scope

• Mixtral of Experts · 2024-01-08 · Checked 2026-10-22

Relations

2024-01-11 · DeepSeekMoE · Method

Fine-grained experts · DeepSeekMoE

Bottleneck

Coarse experts redundantly learn shared knowledge.

Mechanism

Fine-grained experts and isolated shared experts encourage specialization.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

More experts complicate routing and communication.

Primary sources and supported content

Abstract: method and scope

• DeepSeekMoE · 2024-01-11 · Checked 2026-10-22

Relations

• Adoption · DeepSeek-V3 · load balancing

  V3 adopts fine-grained/shared experts and changes balancing.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2024-12-27 · DeepSeek-V3 · load balancing · Model practice

Loss-free balancing · DeepSeek-V3

Bottleneck

Auxiliary balancing may interfere with task learning while communication stays costly.

Mechanism

Expert biases regulate load with less interference from auxiliary balancing loss, retaining fine-grained and shared experts.

Evidence and conditions

The V3 report discloses DeepSeekMoE and MLA, with MLA previously validated in V2; December 2024 is not the invention date of MLA.

New constraints

Bias updates, grouping restrictions and cluster communication require joint tuning.

Primary sources and supported content

Primary source: method and results

• DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

Source revised 2025-02-18

Relations

• Adoption · DeepSeekMoE

  V3 adopts fine-grained/shared experts and changes balancing.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
• Combination · DeepSeek-V3 · MTP

  V3 combines MoE and MTP, which are distinct mechanisms.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
• Cross-track support · DeepSeek-V3 · DualPipe / FP8

  MoE training needs coordinated compute and expert communication.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2025-04-29 · Qwen3 MoE · Model practice

Scaling sparse experts · Qwen3 MoE

Bottleneck

Different deployment budgets require dense and sparse choices.

Mechanism

Qwen3 offers dense and MoE variants, distinguishing total and active parameters.

Evidence and conditions

The 2025 announcement identifies model variants and hybrid thinking. Route labels must refer to specific models and modes.

New constraints

Active parameters do not measure full memory; MoE is not faster at every batch size.

Primary sources and supported content

Primary source: method and results

• Qwen3 · 2025-04-29 · Checked 2026-10-21

Relations

2022-03-04 · InstructGPT / RLHF · Method

Human feedback · OpenAI · InstructGPT

Bottleneck

Next-token pretraining does not directly capture user intent.

Mechanism

Use demonstrations for SFT, comparisons for reward modeling and constrained policy optimization for feedback.

Evidence and conditions

InstructGPT reports this pipeline and evaluations; better human preference scores do not establish universal factual correctness.

New constraints

Preference data and reward models are biased; reward optimization does not ensure truth.

Primary sources and supported content

Primary source: method and results

• Training language models to follow instructions with human feedback · 2022-03-04 · Checked 2026-10-21

Relations

• Alternative · DPO

  A different preference objective, not universal replacement of RLHF.

  • Direct Preference Optimization · 2023-05-29 · Checked 2026-10-21

2022-12-15 · Constitutional AI · Method

Constitutional AI · Constitutional AI

Bottleneck

Item-by-item human safety feedback is costly.

Mechanism

Explicit principles guide critique and revision, followed by AI preference feedback.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Principle choice, evaluator bias and coverage still shape outcomes.

Primary sources and supported content

Abstract: method and scope

• Constitutional AI · 2022-12-15 · Checked 2026-10-22

Relations

2023-05-29 · DPO · Method

Direct preferences · DPO

Bottleneck

Separate reward models and online RL complicate preference training.

Mechanism

Optimize winner/loser probability ratios relative to a reference, without fitting a separate reward model.

Evidence and conditions

DPO derives a preference loss from a KL-regularized objective. It still needs preference pairs and a reference and is an alternative, not a mandatory successor.

New constraints

Offline preference coverage and reference-policy constraints limit exploration.

Primary sources and supported content

Primary source: method and results

• Direct Preference Optimization · 2023-05-29 · Checked 2026-10-21

Relations

• Alternative · InstructGPT / RLHF

  A different preference objective, not universal replacement of RLHF.

  • Direct Preference Optimization · 2023-05-29 · Checked 2026-10-21

2026 · GLM-5.3 · Post-training on a shared base · Model practice

Post-training on a shared base · GLM-5.3

Bottleneck

Capability improvements need not require a new pretrained architecture.

Mechanism

The official description attributes 5.3 gains to post-training on the 5.2 base, separately from the new Flash base.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

This does not establish a particular optimizer, GRPO or distillation.

Primary sources and supported content

README: GLM-5.3 introduction; checked 2026-10-23

• GLM-5.3 / GLM-5.3-Flash official repository · 2026 · Checked 2026-10-23

Relations

2026-09 · Claude Opus 5.5 · Constitution-guided post-training · Model practice

Constitution-guided post-training · Claude Opus 5.5

Bottleneck

Assistants need task capabilities and consistent behavioral principles.

Mechanism

The transparency page describes diverse pretraining followed by substantial post-training aimed at the values in Claude’s constitution.

Evidence and conditions

The Opus 5.5 training description was reviewed; the exact historical CAI recipe is not automatically inherited.

New constraints

The optimizer and reward construction are not disclosed; a specific RLAIF pipeline is not established.

Primary sources and supported content

Claude Opus 5.5: Model architecture and training methodology

• Claude Transparency Hub · 2026-09 · Checked 2026-10-23

Relations

2026-10-12 · Muse Wonder · Pretraining and RL scaling · Model practice

Joint pretraining and RL scaling · Muse Wonder / Full

Bottleneck

Pretraining capacity alone does not account for every long-task capability.

Mechanism

The official post describes scaling both pretraining and RL, without identifying a specific optimizer.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

This does not establish reasoning-specific RL, GRPO or Llama architecture.

Primary sources and supported content

Scaling Toward Personal Superintelligence

• Muse Wonder and Muse Wonder Full · 2026-10-12 · Checked 2026-10-23

Relations

2024-02-05 · DeepSeekMath · GRPO · Method

Group-relative RL · DeepSeekMath

Bottleneck

A value model adds training cost to policy optimization.

Mechanism

Sample a group of outputs per problem and derive relative advantages from group rewards, removing a separate critic.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Little reward variation weakens the signal; policy drift still needs control.

Aᵢ = (rᵢ − mean(r_group)) / std(r_group)
σ = 0 requires explicit handling

Primary sources and supported content

Abstract: method and scope

• DeepSeekMath / GRPO · 2024-02-05 · Checked 2026-10-22

Relations

• Adoption · R1 / R1-Zero

  R1 uses GRPO for verifiable-reward training.

  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · 2025-01-22 · Checked 2026-10-21
• Improvement · DAPO

  Refine clipping, sampling and long-output handling around group-based policy optimization.

  • DAPO · 2025-03-18 · Checked 2026-10-22

2024-09-12 · o1 · reasoning RL · Model practice

Reasoning RL · OpenAI · o1

Bottleneck

Larger pretrained models still fail at multi-step reasoning.

Mechanism

Shape problem solving through RL and study scaling of training and answering compute.

Evidence and conditions

OpenAI discloses the direction and experiments in 2024, not a complete reward or algorithm specification; R1 cannot fill those gaps.

New constraints

Disclosure does not reproduce the full RL recipe; longer thinking adds latency.

Primary sources and supported content

Primary source: method and results

• Learning to reason with LLMs · 2024-09-12 · Checked 2026-10-21

Relations

2025-01-22 · R1 / R1-Zero · Model practice

Verifiable rewards · R1 / R1-Zero

Bottleneck

Human reasoning demonstrations are costly; scalable signals are needed.

Mechanism

Verifiable feedback supports reasoning RL; distinguish direct-RL R1-Zero from cold-start, multistage R1.

Evidence and conditions

The 2025 report describes methods including group-relative rewards. Checking the final outcome does not prove every reasoning step.

New constraints

Verifiable rewards cover limited tasks; R1-Zero readability requires further training.

Primary sources and supported content

Primary source: method and results

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · 2025-01-22 · Checked 2026-10-21

Relations

• Adoption · DeepSeekMath · GRPO

  R1 uses GRPO for verifiable-reward training.

  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · 2025-01-22 · Checked 2026-10-21
• Cross-track support · Test-time compute

  Reasoning training shapes the value of more compute; allocation is a separate problem.

  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · 2025-01-22 · Checked 2026-10-21
  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21

2025-01-22 · Kimi k1.5 · Model practice

Long-chain RL · Kimi k1.5

Bottleneck

Exploration and sample efficiency constrain long-reasoning training.

Mechanism

Scale RL context and policy optimization, then use long-to-short transfer for shorter outputs.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Long trajectories cost more to sample; distilled capability needs separate evaluation.

Primary sources and supported content

Abstract: method and scope

• Kimi k1.5 · 2025-01-22 · Checked 2026-10-22

Relations

• Cross-track support · AReaL

  Long-output RL motivates asynchronous execution; this does not assert k1.5 uses AReaL.

  • Kimi k1.5 · 2025-01-22 · Checked 2026-10-22
  • AReaL · 2025-05-30 · Checked 2026-10-22

2025-03-18 · DAPO · Method

Policy optimization · DAPO

Bottleneck

Entropy collapse, uninformative groups and long outputs undermine RL.

Mechanism

Decoupled clipping, dynamic sampling, token-level loss and overlong reward handling address practical RL failure modes.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Results depend on task rewards and length settings, not a universal recipe.

Primary sources and supported content

Four key techniques / method

• DAPO · 2025-03-18 · Checked 2026-10-22

Relations

• Improvement · DeepSeekMath · GRPO

  Refine clipping, sampling and long-output handling around group-based policy optimization.

  • DAPO · 2025-03-18 · Checked 2026-10-22
• Cross-track support · HybridFlow / verl

  The DAPO implementation is built on verl.

  • DAPO · 2025-03-18 · Checked 2026-10-22

2024-07-31 · Llama 3 data recipe · Model practice

Training data recipe · Meta · Llama 3

Bottleneck

Raw web volume does not measure effective training data.

Mechanism

Deduplication, quality filtering, mixture selection and annealing accompany model scale and compute budgets.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Recipes depend on the data pool; longer training must be weighed against deployment value.

Primary sources and supported content

Pre-training data / data mix

• The Llama 3 Herd of Models · 2024-07-31 · Checked 2026-10-22

Relations

2025-05-14 · Qwen3 · staged pretraining · Model practice

Staged pretraining · Alibaba · Qwen3

Bottleneck

General data, reasoning knowledge and long-context objectives differ.

Mechanism

Staged pretraining changes data mixture and sequence length, followed by thinking/non-thinking post-training integration.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

The report does not expose all data provenance; synthetic volume is not quality.

Primary sources and supported content

Pre-training / post-training

• Qwen3 Technical Report · 2025-05-14 · Checked 2026-10-22

Relations

2026-02-04 · ERNIE 5.0 · Unified multimodal prediction · Model practice

Unified multimodal prediction · ERNIE 5.0

Bottleneck

Understanding and generation modalities have different optimization dynamics.

Mechanism

Serialize modalities into a shared sequence, train with next-group-of-tokens prediction, and route them through shared sparse experts.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

A shared objective still requires modality encoding and data balancing.

Primary sources and supported content

§2 Architecture / §2.1

• ERNIE 5.0 Technical Report · 2026-02-04 · Checked 2026-10-23

Relations

2021-06-17 · LoRA · Method

Low-rank adaptation · LoRA

Bottleneck

Saving full-model updates per task is expensive.

Mechanism

Freeze the base and train low-rank increments, optionally merging them before deployment.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Rank and layer choice constrain adaptation; base inference is not automatically cheaper.

W′ = W + BA, rank(BA) ≤ r

Primary sources and supported content

Abstract: method and scope

• LoRA · 2021-06-17 · Checked 2026-10-22

Relations

• Combination · QLoRA

  Combine low-rank updates with a quantized base.

  • QLoRA · 2023-05-23 · Checked 2026-10-22

2023-05-23 · QLoRA · Method

Quantized adaptation · QLoRA

Bottleneck

The base still occupies memory when only adapters are trained.

Mechanism

An NF4 base, double quantization and paged optimizers reduce fine-tuning memory alongside low-rank updates.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Quantization error, dequantization overhead and task quality must be measured together.

Primary sources and supported content

Abstract: method and scope

• QLoRA · 2023-05-23 · Checked 2026-10-22

Relations

• Combination · LoRA

  Combine low-rank updates with a quantized base.

  • QLoRA · 2023-05-23 · Checked 2026-10-22

2023-12-01 · Mamba · Method

Selective state space · Mamba

Bottleneck

Full-attention cache grows with sequence length.

Mechanism

Input-dependent selective state-space updates and hardware-aware scans compress history into recurrent state.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Fixed state is not exact random access to arbitrary history.

Primary sources and supported content

Abstract: method and scope

• Mamba · 2023-12-01 · Checked 2026-10-22

Relations

2024-12-09 · Gated DeltaNet · Method

Gated recurrent state · Gated DeltaNet

Bottleneck

Linear attention needs more selective memory updates.

Mechanism

Combine gated forgetting with delta-rule updates to control retention and writing.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Compressed state can lose information; parallel training has implementation costs.

Primary sources and supported content

Abstract: method and scope

• Gated Delta Networks · 2024-12-09 · Checked 2026-10-22

Relations

• Adoption · Qwen3-Next

  Qwen3-Next discloses Gated DeltaNet use.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
• Improvement · Kimi K3 · KDA with global attention

  KDA adds channel-wise forgetting to delta recurrence and combines it with global layers in K3.

  • Kimi K3: Open Frontier Intelligence · 2026-07-27 · Checked 2026-10-23

2025-01-14 · MiniMax-01 · Model practice

Hybrid attention · MiniMax-01

Bottleneck

Very long contexts are costly to train and serve.

Mechanism

Mix Lightning Attention with softmax attention for linear state updates and global interaction.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

The mixture affects retrieval quality and cache cost.

Primary sources and supported content

Abstract: method and scope

• MiniMax-01 · 2025-01-14 · Checked 2026-10-22

Relations

2025-06-16 · MiniMax-M1 · Model practice

Hybrid reasoning RL · MiniMax-M1

Bottleneck

Long reasoning adds cost on top of long context.

Mechanism

Use a hybrid-attention MoE for long reasoning with math and software-environment RL.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

A long window does not guarantee use; reported speed depends on output length and baseline.

Primary sources and supported content

Abstract: method and scope

• MiniMax-M1 · 2025-06-16 · Checked 2026-10-22

Relations

2025-09-10 · Qwen3-Next · Model practice

Hybrid attention + MoE · Qwen3-Next

Bottleneck

Dense attention and expert compute constrain efficient long context.

Mechanism

Combine Gated DeltaNet and gated attention with highly sparse MoE and multi-token prediction.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Runtime support for hybrid state and MTP is required; ablations distinguish component gains.

Primary sources and supported content

Architecture / training / MTP

• Qwen3-Next · 2025-09-10 · Checked 2026-10-22

Relations

• Adoption · Gated DeltaNet

  Qwen3-Next discloses Gated DeltaNet use.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
• Combination · Qwen3.5

  Both disclosures combine hybrid attention and sparse experts; this does not infer training lineage.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
  • Qwen3.5 · 2026-02-15 · Checked 2026-10-22
• Cross-track support · DeepSeek-V3 · MTP

  Both model disclosures use MTP, connecting training objectives and drafting.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2026 · GLM-5.3-Flash · Sparse / linear hybrid with mHC · Model practice

Sparse / linear hybrid with mHC · GLM-5.3-Flash

Bottleneck

Efficient serving must balance state cost and attention capacity.

Mechanism

The official description gives Flash a new base combining sparse and linear attention with mHC.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

It is not simply a smaller 5.3; GLM-5 infrastructure cannot be assumed to carry over.

Primary sources and supported content

README: GLM-5.3-Flash; checked 2026-10-23

• GLM-5.3 / GLM-5.3-Flash official repository · 2026 · Checked 2026-10-23

Relations

2026 · Qwen3.8-2.4T-A95B · Gated recurrence / attention with MoE · Model practice

Gated recurrence / attention with MoE · Qwen3.8-2.4T-A95B

Bottleneck

A Max-class open model must balance capacity and per-token compute.

Mechanism

The card specifies interleaved Gated DeltaNet and Gated Attention, MoE, and multi-step MTP training.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Bound to the open weights; MTP training alone does not establish speculative serving.

Primary sources and supported content

Model Overview

• Qwen3.8-2.4T-A95B model card · 2026 · Checked 2026-10-23

Relations

2026-02-15 · Qwen3.5 · Model practice

Scaling hybrid models · Qwen3.5

Bottleneck

Hybrid-state models must also process visual inputs.

Mechanism

Native vision-language training combines with hybrid attention and sparse experts.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Text scores alone do not establish multimodal or long-context capability.

Primary sources and supported content

Official architecture and multimodal-training description

• Qwen3.5 · 2026-02-15 · Checked 2026-10-22

Model overview / hybrid layout / vision-language foundation

• Qwen3.5 model card · Date not stated · Checked 2026-10-22

Relations

• Combination · Qwen3-Next

  Both disclosures combine hybrid attention and sparse experts; this does not infer training lineage.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
  • Qwen3.5 · 2026-02-15 · Checked 2026-10-22
• Cross-track support · Dynamo · EPD

  Multimodal models introduce encoding load; separation needs measurement and does not imply Qwen uses Dynamo.

  • Qwen3.5 · 2026-02-15 · Checked 2026-10-22
  • When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21

2026-04-14 · Nemotron 3 Super · Model practice

Hybrid expert models · Nemotron 3 Super

Bottleneck

Long agent trajectories require lower cache and expert costs.

Mechanism

Mix Mamba, attention and LatentMoE, with MTP for native speculative decoding.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Hybrid structure and speculation require kernels; active parameter count alone does not predict gains.

Primary sources and supported content

Abstract: method and scope

• Nemotron 3 Super · 2026-04-14 · Checked 2026-10-22

Relations

• Cross-track support · Nemotron 3 Super · NVFP4

  Architecture and low-precision pretraining in the same model.

  • Nemotron 3 Super · 2026-04-14 · Checked 2026-10-22

2026-07-27 · Kimi K3 · KDA with global attention · Model practice

KDA with global attention · Kimi K3

Bottleneck

Long sequences need efficient state updates and global retrieval.

Mechanism

Each block combines three KDA layers with one Gated MLA layer. KDA adds channel-wise forgetting to delta recurrence.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Global and recurrent layers need different kernels; the ratio is specific to K3.

Primary sources and supported content

§2.1 Hybrid Attention / §2.1.1 KDA

• Kimi K3: Open Frontier Intelligence · 2026-07-27 · Checked 2026-10-23

Source revised 2026-08-07

Relations

• Improvement · Gated DeltaNet

  KDA adds channel-wise forgetting to delta recurrence and combines it with global layers in K3.

  • Kimi K3: Open Frontier Intelligence · 2026-07-27 · Checked 2026-10-23

2025-07-28 · Kimi K2 · MuonClip · Model practice

Optimizer stability · Kimi K2 · MuonClip

Bottleneck

Large MoE training with Muon needs control of attention outliers.

Mechanism

QK clipping constrains attention-logit growth alongside Muon for stable large-scale training.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

This is a reported stabilization recipe, not evidence every model should replace Adam.

Primary sources and supported content

MuonClip / pre-training

• Kimi K2 · 2025-07-28 · Checked 2026-10-22

Source revised 2026-02-03

Relations

2025-12-31 · mHC · Method

Constrained connections · mHC

Bottleneck

Unconstrained mixing of wider residual streams can destabilize signal propagation.

Mechanism

Constrain residual mixing to a manifold to retain stable propagation with richer representations.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Wider streams and projection add cost; the first publication was in late 2025.

Primary sources and supported content

Abstract: method and scope

• mHC: Manifold-Constrained Hyper-Connections · 2025-12-31 · Checked 2026-10-22

Source revised 2026-01-05

Relations

• Adoption · DeepSeek-V4 · mHC adoption

  V4 model materials explicitly disclose mHC.

  • DeepSeek-V4 model card · Date not stated · Checked 2026-10-22

2026-01-12 · Engram · Method

Conditional memory · Engram

Bottleneck

Retrieving repeated patterns consumes neural compute capacity.

Mechanism

Conditional N-gram lookup memory complements conditional compute; deterministic addresses enable host-memory prefetch.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Memory capacity, bandwidth and allocation trade off; this does not establish V4 adoption.

Primary sources and supported content

Abstract: method and scope

• Conditional Memory via Scalable Lookup: Engram · 2026-01-12 · Checked 2026-10-22

Relations

2026-04-24 · DeepSeek-V4 · mHC adoption · Model practice

mHC model adoption · DeepSeek-V4

Bottleneck

Architectural research needs validation in full-model training.

Mechanism

Official model materials disclose mHC and Muon; method origins and model adoption are separate events.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Undisclosed components cannot be inferred from other research by the same company.

Primary sources and supported content

Abstract: method and scope

• DeepSeek-V4 release · 2026-04-24 · Checked 2026-10-22

Architecture / optimization

• DeepSeek-V4 model card · Date not stated · Checked 2026-10-22

Relations

• Adoption · mHC

  V4 model materials explicitly disclose mHC.

  • DeepSeek-V4 model card · Date not stated · Checked 2026-10-22

2025-04-16 · o3 / o4-mini · tool training · Model practice

Tool-use training · OpenAI · o3 / o4-mini

Bottleneck

Answering does not imply knowing when to use tools.

Mechanism

RL trains tool selection and use, interleaving reasoning with search, code and visual operations.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

The announcement does not disclose the full algorithm; changing tool environments affects outcomes.

Primary sources and supported content

Learning to use tools

• OpenAI o3 and o4-mini · 2025-04-16 · Checked 2026-10-22

Relations

• Cross-track support · o3 / o4-mini · interleaved tools

  Tool training supports inference-time tool use.

  • OpenAI o3 and o4-mini · 2025-04-16 · Checked 2026-10-22

2025-07-28 · Kimi K2 · agent environments · Model practice

Agent environments · Kimi K2

Bottleneck

Real tool trajectories are limited and costly to collect.

Mechanism

Synthesize tool-use data and apply joint RL in real and synthetic environments.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Synthetic rewards can mismatch real tasks.

Primary sources and supported content

Agentic data synthesis / RL

• Kimi K2 · 2025-07-28 · Checked 2026-10-22

Source revised 2026-02-03

Relations

2025-12-02 · DeepSeek-V3.2 · agent RL · Model practice

Agent RL · DeepSeek-V3.2

Bottleneck

Complex tool tasks lack scalable training signals.

Mechanism

Scale RL and agent-task synthesis for tool use during reasoning.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Task-generator and verifier quality limit coverage.

Primary sources and supported content

Abstract: method and scope

• DeepSeek-V3.2 · 2025-12-02 · Checked 2026-10-22

Relations

2026-02-02 · Kimi K2.5 · PARL · Model practice

Parallel agent RL · Kimi K2.5 · PARL

Bottleneck

Serial agents do not automatically learn useful parallel delegation.

Mechanism

Train an orchestrator over frozen subagents with staged rewards to counter serial collapse and optimize the critical path.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Sparse delayed feedback and dependencies limit parallelism; call count alone is a poor reward.

Primary sources and supported content

Abstract: method and scope

• Kimi K2.5 · 2026-02-02 · Checked 2026-10-22

Source revised 2026-08-07

Parallel Agent Reinforcement Learning / critical steps

• Kimi K2.5: Parallel Agent Reinforcement Learning · Date not stated · Checked 2026-10-22

Relations

• Cross-track support · Kimi K2.5 · parallel agents

  Coordination training shapes inference-time orchestration.

  • Kimi K2.5 · 2026-02-02 · Checked 2026-10-22
  • Kimi K2.5: Parallel Agent Reinforcement Learning · Date not stated · Checked 2026-10-22

2018-10-11 · BERT · Method

Masked pretraining · Google · BERT

Bottleneck

Causal language models cannot jointly use left and right context.

Mechanism

Masked-token prediction uses left and right context at every layer, with task fine-tuning transferring representations.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Useful for representation and understanding, not direct autoregressive generation.

Primary sources and supported content

Abstract: method and scope

• BERT · 2018-10-11 · Checked 2026-10-22

Relations

• Alternative · Transformer

  Bidirectional encoding and causal generation use different attention visibility and objectives.

  • BERT · 2018-10-11 · Checked 2026-10-22
  • Attention Is All You Need · 2017-06-12 · Checked 2026-10-21
• Alternative · T5

  Masked representation learning and denoising text-to-text use different pretraining interfaces.

  • BERT · 2018-10-11 · Checked 2026-10-22
  • T5 · 2019-10-23 · Checked 2026-10-22

2019-10-23 · T5 · Method

Text-to-text learning · Google · T5

Bottleneck

Tasks use different input and output forms.

Mechanism

Unify language tasks as text-to-text, comparing objectives, data and architectures with a denoising encoder–decoder.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Two stacks complicate deployment and compute organization.

Primary sources and supported content

Abstract: method and scope

• T5 · 2019-10-23 · Checked 2026-10-22

Relations

• Alternative · BERT

  Masked representation learning and denoising text-to-text use different pretraining interfaces.

  • BERT · 2018-10-11 · Checked 2026-10-22
  • T5 · 2019-10-23 · Checked 2026-10-22

2019-11-05 · MQA · Method

Shared KV heads · MQA

Bottleneck

Independent KV per query head makes decode bandwidth and cache expensive.

Mechanism

Share one K/V set across query heads to reduce incremental-decoding cache traffic.

Evidence and conditions

The 2019 MQA work reduces bandwidth demand through shared KV; quality and throughput remain model- and implementation-dependent.

New constraints

Aggressive sharing can hurt quality and needs training adaptation.

Primary sources and supported content

Primary source: method and results

• Fast Transformer Decoding: One Write-Head is All You Need · 2019-11-05 · Checked 2026-10-21

Relations

• Alternative · GQA

  Grouping trades shared KV against multi-head expressiveness.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21

2023-05-22 · GQA · Method

Grouped KV heads · GQA

Bottleneck

A middle ground is needed between MQA cache savings and MHA quality.

Mechanism

Share K/V within query-head groups, choosing a memory/quality tradeoff between MHA and MQA.

Evidence and conditions

The 2023 GQA paper studies model conversion and training; cache-payload ratios are not end-to-end speed ratios.

New constraints

Group count controls the tradeoff; conversion needs uptraining.

KV bytes ≈ 2 × B × L × T × H_KV × d × bytes/element

Primary sources and supported content

Primary source: method and results

• GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21

Relations

• Alternative · MQA

  Grouping trades shared KV against multi-head expressiveness.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
• Alternative · DeepSeek-V2 · MLA

  Shared heads and latent compression are different KV representations.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
  • DeepSeek-V2 · 2024-05-07 · Checked 2026-10-22
• Cross-track support · PagedAttention / vLLM

  Representation compression and memory paging are complementary.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
  • Efficient Memory Management for Large Language Model Serving with PagedAttention · 2023-09-12 · Checked 2026-10-21

2024-05-07 · DeepSeek-V2 · MLA · Model practice

Latent KV compression · DeepSeek-V2 · MLA

Bottleneck

KV cache limits concurrency and long context.

Mechanism

Jointly compress KV into a latent vector; decoupled positional encoding and weight absorption reduce decode cache.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

It needs matching training and kernels, not a drop-in cache replacement.

Primary sources and supported content

Abstract: method and scope

• DeepSeek-V2 · 2024-05-07 · Checked 2026-10-22

Relations

• Alternative · GQA

  Shared heads and latent compression are different KV representations.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
  • DeepSeek-V2 · 2024-05-07 · Checked 2026-10-22
• Adoption · DeepSeek-V3 · MLA adoption

  V3 adopts MLA already disclosed in V2.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2024-12-27 · DeepSeek-V3 · MLA adoption · Model practice

MLA adoption · DeepSeek-V3

Bottleneck

Large-scale models still need lower KV memory.

Mechanism

V3 adopts MLA introduced in V2; this is model adoption, with the origin recorded in the May 2024 V2 node.

Evidence and conditions

The V3 report discloses DeepSeekMoE and MLA, with MLA previously validated in V2; December 2024 is not the invention date of MLA.

New constraints

V3 is an MLA adoption example, not its origin.

Primary sources and supported content

Primary source: method and results

• DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

Source revised 2025-02-18

Relations

• Adoption · DeepSeek-V2 · MLA

  V3 adopts MLA already disclosed in V2.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2022-11-30 · Speculative decoding · Method

Draft and verify · Speculative decoding

Bottleneck

Serial target decoding repeatedly reads weights.

Mechanism

Draft candidates, verify them in parallel, and use acceptance/rejection with residual sampling to recover the target distribution.

Evidence and conditions

First disclosed in 2022 and published at ICML 2023. Distribution preservation depends on the full correction algorithm and assumptions.

New constraints

Draft cost and rejection can erase gains; distribution guarantees require correction.

Primary sources and supported content

Primary source: method and results

• Fast Inference from Transformers via Speculative Decoding · 2022-11-30 · Checked 2026-10-21

Relations

• Improvement · EAGLE-3

  Retain target verification while improving draft training and features.

  • EAGLE-3 · 2025-03-03 · Checked 2026-10-22

2024 · AI Overviews · Engineering

Speculation in use · Google AI Overviews

Bottleneck

Production generation needs lower user wait time.

Mechanism

Google discloses speculative decoding in AI Overviews to reduce expensive generation steps.

Evidence and conditions

This is a 2024 adoption disclosure, not the method’s invention, and does not promise identical gains for all requests.

New constraints

The product note does not disclose all acceptance policies or implementation details.

Primary sources and supported content

Primary source: method and results

• Looking back at speculative decoding · 2024 · Checked 2026-10-21

Relations

2024-01-19 · Medusa · Method

Multi-head drafting · Medusa

Bottleneck

A separate draft model adds deployment and scheduling cost.

Mechanism

Add decoding heads to the target model and verify a tree of candidate future tokens.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Extra heads need training; acceptance policies have different distribution guarantees.

Primary sources and supported content

Abstract: method and scope

• Medusa · 2024-01-19 · Checked 2026-10-22

Relations

2024-12-27 · DeepSeek-V3 · MTP · Model practice

Multi-token prediction · DeepSeek-V3 · MTP

Bottleneck

A single-step objective does not directly provide cheap multi-step drafts.

Mechanism

Train auxiliary multi-token prediction modules that can draft tokens for target verification.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Training benefits and inference acceleration are distinct claims.

Primary sources and supported content

Multi-token prediction

• DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

Source revised 2025-02-18

Relations

• Combination · DeepSeek-V3 · load balancing

  V3 combines MoE and MTP, which are distinct mechanisms.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
• Cross-track support · Qwen3-Next

  Both model disclosures use MTP, connecting training objectives and drafting.

  • Qwen3-Next · 2025-09-10 · Checked 2026-10-22
  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

2025-03-03 · EAGLE-3 · Method

Feature-based drafts · EAGLE-3

Bottleneck

Feature-prediction constraints limit gains from more draft-training data.

Mechanism

Replace feature prediction with direct token prediction, using multi-layer target features and training-time testing.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Speed depends on acceptance rate, draft cost and target workload.

Primary sources and supported content

Abstract: method and scope

• EAGLE-3 · 2025-03-03 · Checked 2026-10-22

Relations

• Improvement · Speculative decoding

  Retain target verification while improving draft training and features.

  • EAGLE-3 · 2025-03-03 · Checked 2026-10-22
• Alternative · DFlash

  Compare autoregressive and block-diffusion drafts without claiming direct lineage.

  • EAGLE-3 · 2025-03-03 · Checked 2026-10-22
  • DFlash · 2026-02-05 · Checked 2026-10-22

2026-02-05 · DFlash · Method

Block diffusion drafts · DFlash

Bottleneck

An autoregressive drafter still generates tokens serially.

Mechanism

Condition on target context features, draft a token block with diffusion, then verify with the autoregressive target.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Losslessness relies on verification and correction, not replacement of the target by diffusion.

Primary sources and supported content

Abstract: method and scope

• DFlash · 2026-02-05 · Checked 2026-10-22

Source revised 2026-05-28

Relations

• Alternative · EAGLE-3

  Compare autoregressive and block-diffusion drafts without claiming direct lineage.

  • EAGLE-3 · 2025-03-03 · Checked 2026-10-22
  • DFlash · 2026-02-05 · Checked 2026-10-22

2022-01-28 · Chain-of-Thought · Method

Chain-of-thought · Chain-of-Thought

Bottleneck

Direct answers do not expose a multi-step solving process.

Mechanism

Reasoning exemplars elicit intermediate steps, moving computation into the output sequence.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Steps can be wrong or unfaithful; longer output is not evidence of correctness.

Primary sources and supported content

Abstract: method and scope

• Chain-of-Thought Prompting · 2022-01-28 · Checked 2026-10-22

Relations

• Combination · Self-Consistency

  Combine multiple chains with answer aggregation.

  • Self-Consistency · 2022-03-21 · Checked 2026-10-22

2022-03-21 · Self-Consistency · Method

Multi-chain aggregation · Self-Consistency

Bottleneck

A single reasoning path can make a local error.

Mechanism

Sample multiple reasoning paths and aggregate final answers for robustness.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Correlated samples, aggregation and total token cost limit gains.

answer* = argmax_a count(sampled answers = a)

Primary sources and supported content

Abstract: method and scope

• Self-Consistency · 2022-03-21 · Checked 2026-10-22

Relations

• Combination · Chain-of-Thought

  Combine multiple chains with answer aggregation.

  • Self-Consistency · 2022-03-21 · Checked 2026-10-22

2023-05-31 · Process supervision · Method

Process supervision · Process supervision

Bottleneck

Outcome-only judgments cannot locate intermediate errors.

Mechanism

Supervise reasoning steps and train a process reward model to score candidates.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Step labels are costly and evaluators can be exploited.

Primary sources and supported content

Abstract: method and scope

• Let’s Verify Step by Step · 2023-05-31 · Checked 2026-10-22

Relations

• Combination · Test-time compute

  A verifier is one component in compute allocation.

  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21

2024-08-06 · Test-time compute · Method

Compute allocation · Test-time compute

Bottleneck

Hard problems may benefit from more search and verification.

Mechanism

Allocate answering budgets among longer trajectories, search and verification according to difficulty.

Evidence and conditions

The 2024 study examines optimal inference-compute allocation, conditional on models, tasks and verifiers.

New constraints

More tokens do not guarantee correctness; measure cost, latency and stopping behavior.

Primary sources and supported content

Primary source: method and results

• Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21

Relations

• Cross-track support · R1 / R1-Zero

  Reasoning training shapes the value of more compute; allocation is a separate problem.

  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · 2025-01-22 · Checked 2026-10-21
  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21
• Combination · Process supervision

  A verifier is one component in compute allocation.

  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21
• Combination · Gemini 2.5 · thinking budget

  Compare allocation research with a product budget interface without asserting adoption of the paper.

  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21
  • Gemini 2.5: thinking budgets · 2025-05-20 · Checked 2026-10-22

2025-02-24 · Extended thinking · Model practice

Extended thinking · Claude 3.7 Sonnet

Bottleneck

Hard problems may benefit from more search and verification.

Mechanism

Claude 3.7 offers standard responses and extended thinking with an adjustable budget.

Evidence and conditions

The 2025 announcement discloses product behavior, not an internal search or reward algorithm.

New constraints

More tokens do not guarantee correctness; measure cost, latency and stopping behavior.

Primary sources and supported content

Primary source: method and results

• Claude 3.7 Sonnet and Claude Code · 2025-02-24 · Checked 2026-10-21

Relations

2025-04-29 · Hybrid thinking · Model practice

Hybrid thinking · Qwen3

Bottleneck

Hard problems may benefit from more search and verification.

Mechanism

Qwen3 offers thinking and non-thinking modes, making answering budget a selectable dimension.

Evidence and conditions

The 2025 announcement identifies model variants and hybrid thinking. Route labels must refer to specific models and modes.

New constraints

More tokens do not guarantee correctness; measure cost, latency and stopping behavior.

Primary sources and supported content

Primary source: method and results

• Qwen3 · 2025-04-29 · Checked 2026-10-21

Relations

• Alternative · Claude Opus 4.6 · adaptive thinking

  Explicit thinking modes/budgets and adaptive thinking are different controls.

  • Qwen3 · 2025-04-29 · Checked 2026-10-21
  • Claude Opus 4.6: adaptive thinking · 2026-02-05 · Checked 2026-10-22

2025-05-20 · Gemini 2.5 · thinking budget · Model practice

Thinking budgets · Gemini 2.5

Bottleneck

Tasks derive different value from thinking compute.

Mechanism

Expose thinking-budget control to trade off quality, cost and latency.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Budget control is an interface, not full disclosure of the internal solver.

Primary sources and supported content

Thinking budgets / developer experience

• Gemini 2.5: thinking budgets · 2025-05-20 · Checked 2026-10-22

Relations

• Combination · Test-time compute

  Compare allocation research with a product budget interface without asserting adoption of the paper.

  • Scaling LLM Test-Time Compute Optimally · 2024-08-06 · Checked 2026-10-21
  • Gemini 2.5: thinking budgets · 2025-05-20 · Checked 2026-10-22

2026-02-05 · Claude Opus 4.6 · adaptive thinking · Model practice

Adaptive thinking · Claude Opus 4.6

Bottleneck

Fixed thinking length allocates compute poorly across task difficulty.

Mechanism

Choose when and how deeply to think from context, with effort controls trading off quality, cost and speed.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

The full budget policy is undisclosed; this is not a reproducible algorithm or guaranteed optimal stopping.

Primary sources and supported content

Product and API updates / adaptive thinking

• Claude Opus 4.6: adaptive thinking · 2026-02-05 · Checked 2026-10-22

Relations

• Alternative · Hybrid thinking

  Explicit thinking modes/budgets and adaptive thinking are different controls.

  • Qwen3 · 2025-04-29 · Checked 2026-10-21
  • Claude Opus 4.6: adaptive thinking · 2026-02-05 · Checked 2026-10-22

2026-09-03 · GPT-6 Astra · Learned internal reasoning · Model practice

Learned internal reasoning · GPT-6 Astra / GPT-6.1 Sol

Bottleneck

Complex tasks require intermediate reasoning before the final answer.

Mechanism

The card describes RL-trained internal reasoning, strategy exploration and error recognition before answering, rather than only a prompting technique.

Evidence and conditions

Explicitly described in §2; reasoning traces are not guaranteed to expose all internal computation.

New constraints

Search algorithms and compute allocation are not disclosed; tree search or multi-model cooperation cannot be inferred.

Primary sources and supported content

§2 Model Data and Training

• GPT-6 Astra System Card · 2026-09-03 · Checked 2026-10-23

Relations

2021-04-20 · RoPE · Method

Rotary positions · RoPE

Bottleneck

Attention needs an encoding that expresses relative position.

Mechanism

Rotate query/key positional representations so dot products encode relative position.

Evidence and conditions

RoFormer introduces RoPE in 2021. Positional encoding does not guarantee retrieval or reasoning at unseen lengths.

New constraints

RoPE neither removes quadratic attention nor guarantees length extrapolation.

Primary sources and supported content

Primary source: method and results

• RoFormer: Enhanced Transformer with Rotary Position Embedding · 2021-04-20 · Checked 2026-10-21

Relations

• Cross-track support · DeepSpeed Ulysses

  Positional extensibility and distributed execution address different bottlenecks.

  • RoFormer: Enhanced Transformer with Rotary Position Embedding · 2021-04-20 · Checked 2026-10-21
  • DeepSpeed Ulysses · 2023-09-25 · Checked 2026-10-22

2023-07-06 · Lost in the Middle · Method

Context utilization · Lost in the Middle

Bottleneck

A larger window does not ensure use of information in its middle.

Mechanism

Place evidence at different context positions to test effective use separately from capacity.

Evidence and conditions

The 2023 work finds positional sensitivity in its tested models and tasks; this is not a universal result for later models.

New constraints

This is an evaluation finding, not a new positional algorithm.

Primary sources and supported content

Primary source: method and results

• Lost in the Middle: How Language Models Use Long Contexts · 2023-07-06 · Checked 2026-10-21

Relations

2023-08-31 · YaRN · Method

Context extension · YaRN

Bottleneck

Positional extrapolation beyond training length is unstable.

Mechanism

Adjust RoPE interpolation by frequency and attention temperature to extend context with limited long-sequence fine-tuning.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

A larger window is not reliable retrieval; attention still needs system optimization.

Primary sources and supported content

Abstract: method and scope

• YaRN · 2023-08-31 · Checked 2026-10-22

Relations

2022-10-31 · GPTQ · Method

Weight quantization · GPTQ

Bottleneck

Layerwise weight quantization accumulates output error.

Mechanism

Approximate second-order information compensates columnwise quantization error for post-training weight quantization.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Calibration data and matching low-bit matrix kernels are required.

Primary sources and supported content

Abstract: method and scope

• GPTQ · 2022-10-31 · Checked 2026-10-22

Relations

2022-11-18 · SmoothQuant · Method

Activation smoothing · SmoothQuant

Bottleneck

Activation outliers complicate low-bit activation quantization.

Mechanism

Equivalent scaling shifts quantization difficulty from activations to weights for W8A8.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Scaling needs calibration; weight and activation accuracy must be evaluated together.

Primary sources and supported content

Abstract: method and scope

• SmoothQuant · 2022-11-18 · Checked 2026-10-22

Relations

2023-06-01 · AWQ · Method

Activation-aware weights · AWQ

Bottleneck

Uniform low-bit quantization can harm salient weight channels.

Mechanism

Use activation information to scale sensitive weights and control low-bit weight-quantization errors.

Evidence and conditions

AWQ targets weight quantization. KV/activation precision, kernel support and evaluation tasks need separate specification.

New constraints

Representative activation calibration and kernels are needed; compression is not speedup.

Primary sources and supported content

Primary source: method and results

• AWQ: Activation-aware Weight Quantization · 2023-06-01 · Checked 2026-10-21

Relations

2024-02-05 · KIVI · KV quantization · Method

KV quantization · KIVI

Bottleneck

Long-context KV remains large after weight quantization.

Mechanism

Use different quantization axes for K and V and retain a recent high-precision residual window.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Scale metadata and dequantization cost prevent using the ideal byte ratio as the actual gain.

Primary sources and supported content

Abstract: method and scope

• KIVI · 2024-02-05 · Checked 2026-10-22

Relations

2025-12-02 · DeepSeek-V3.2 · DSA · Model practice

Sparse attention · DeepSeek-V3.2

Bottleneck

Reading all attention history is costly in long contexts.

Mechanism

A lightweight indexer selects tokens and main attention reads the selected history.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Indexer cost, selection quality and kernels jointly determine gains.

Selected attention work ≈ O(Tk·d), k ≪ T
Excludes indexer / selection cost

Primary sources and supported content

Abstract: method and scope

• DeepSeek-V3.2 · 2025-12-02 · Checked 2026-10-22

Relations

• Improvement · DeepSeek-V4 · CSA / HCA

  The model route adds KV compression to sparse selection.

  • DeepSeek-V3.2 · 2025-12-02 · Checked 2026-10-22
  • DeepSeek-V4 model card · Date not stated · Checked 2026-10-22
• Alternative · MiniMax M3 · Block sparse attention

  The official comparison presents block selection and KV-outer execution as an alternative sparse route.

  • MiniMax M3 · 2026-06-01 · Checked 2026-10-23

2026-04-24 · DeepSeek-V4 · CSA / HCA · Model practice

Compressed attention · DeepSeek-V4

Bottleneck

Sparse reads alone do not fully control stored KV.

Mechanism

CSA combines KV compression and sparse selection; HCA applies dense attention over more heavily compressed state.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Compression changes retained information and kernel needs; it is not lossless paging.

Primary sources and supported content

Abstract: method and scope

• DeepSeek-V4 release · 2026-04-24 · Checked 2026-10-22

Hybrid attention architecture

• DeepSeek-V4 model card · Date not stated · Checked 2026-10-22

Relations

• Improvement · DeepSeek-V3.2 · DSA

  The model route adds KV compression to sparse selection.

  • DeepSeek-V3.2 · 2025-12-02 · Checked 2026-10-22
  • DeepSeek-V4 model card · Date not stated · Checked 2026-10-22
• Improvement · DeepSeek V4.1-Flash · Cross-layer KV and index reuse

  CSA2 adds cross-layer KV and index reuse to compressed context; this is not a weight-inheritance claim.

  • DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression · 2026-09-17 · Checked 2026-10-23

2026-06-01 · MiniMax M3 · Block sparse attention · Model practice

Block sparse attention · MiniMax M3

Bottleneck

Full attention is costly at million-token context.

Mechanism

MSA preselects KV blocks; its kernel gathers matching queries around each KV block to reduce repeated reads.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Benefits depend on head configuration and context; M3.1 Flash adoption is not established.

Primary sources and supported content

MSA: Architectural Innovation Enables Context Scaling

• MiniMax M3 · 2026-06-01 · Checked 2026-10-23

Relations

• Alternative · DeepSeek-V3.2 · DSA

  The official comparison presents block selection and KV-outer execution as an alternative sparse route.

  • MiniMax M3 · 2026-06-01 · Checked 2026-10-23

2026-09-17 · DeepSeek V4.1-Flash · Cross-layer KV and index reuse · Model practice

Cross-layer KV and index reuse · DeepSeek V4.1-Flash

Bottleneck

Long-horizon agents stress KV storage and transfer.

Mechanism

CSA2 uses Full, Reindex and Reuse layers, sharing main KV and index keys while recomputing or reusing Top-K selections.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Sharing and reindexing trade accuracy against cost; compression does not create unlimited context.

Primary sources and supported content

§2.3–2.3.1 CSA2

• DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression · 2026-09-17 · Checked 2026-10-23

Relations

• Improvement · DeepSeek-V4 · CSA / HCA

  CSA2 adds cross-layer KV and index reuse to compressed context; this is not a weight-inheritance claim.

  • DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression · 2026-09-17 · Checked 2026-10-23

2025-04-16 · o3 / o4-mini · interleaved tools · Model practice

Interleaved tools · OpenAI · o3 / o4-mini

Bottleneck

Text-only reasoning cannot read external state or execute computation.

Mechanism

Select tools during reasoning, incorporate results and continue; the budget includes tool calls.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

End-to-end latency includes tool waits; tool-assisted and model-only evaluations differ.

Primary sources and supported content

Tool use during reasoning

• OpenAI o3 and o4-mini · 2025-04-16 · Checked 2026-10-22

Relations

• Cross-track support · o3 / o4-mini · tool training

  Tool training supports inference-time tool use.

  • OpenAI o3 and o4-mini · 2025-04-16 · Checked 2026-10-22

2026-02-02 · Kimi K2.5 · parallel agents · Model practice

Parallel agents · Kimi K2.5

Bottleneck

Serial execution accumulates latency for decomposable work.

Mechanism

An orchestrator delegates concurrent subtasks and synthesizes results; evaluation considers the longest dependency path.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Concurrency can reduce wall time while increasing tokens; dependency chains remain serial.

wall time ≥ longest dependency path
parallel wall time ≠ total token cost

Primary sources and supported content

Abstract: method and scope

• Kimi K2.5 · 2026-02-02 · Checked 2026-10-22

Source revised 2026-08-07

Agent swarm / critical steps

• Kimi K2.5: Parallel Agent Reinforcement Learning · Date not stated · Checked 2026-10-22

Relations

• Cross-track support · Kimi K2.5 · PARL

  Coordination training shapes inference-time orchestration.

  • Kimi K2.5 · 2026-02-02 · Checked 2026-10-22
  • Kimi K2.5: Parallel Agent Reinforcement Learning · Date not stated · Checked 2026-10-22

2019-10-04 · ZeRO · Method

Training-state sharding · ZeRO

Bottleneck

Data parallelism replicates training states on every device.

Mechanism

Progressively shard optimizer states, gradients and parameters to reduce data-parallel redundancy.

Evidence and conditions

The 2019 ZeRO paper presents state partitioning; ideal state savings exclude activations, communication buffers and peak gather costs.

New constraints

Sharding saves memory but adds collectives; stages depend on the network.

Primary sources and supported content

Primary source: method and results

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models · 2019-10-04 · Checked 2026-10-21

Relations

2020-02-13 · DeepSpeed · Engineering

Sharding in practice · Microsoft · DeepSpeed

Bottleneck

State sharding needs an operational training runtime.

Mechanism

The DeepSpeed release brings ZeRO state optimization into a training system, balancing memory and communication.

Evidence and conditions

Microsoft’s 2020 announcement is project-adoption evidence; the ZeRO paper remains dated 2019.

New constraints

DeepSpeed is engineering adoption, not the first ZeRO paper date.

Primary sources and supported content

Primary source: method and results

• ZeRO & DeepSpeed release · 2020-02-13 · Checked 2026-10-21

Relations

2018-11-16 · GPipe · Method

Pipeline parallelism · GPipe

Bottleneck

A single device cannot hold the model.

Mechanism

Partition layers into a pipeline and split batches into microbatches to amortize bubbles.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Stage imbalance and microbatch count affect utilization and memory.

Primary sources and supported content

Abstract: method and scope

• GPipe · 2018-11-16 · Checked 2026-10-22

Relations

2019-09-17 · Megatron-LM · Method

Tensor parallelism · Megatron-LM

Bottleneck

Large matrices and models exceed a single GPU.

Mechanism

Partition tensor operations within Transformer layers across GPUs.

Evidence and conditions

Megatron-LM in 2019 studies intra-layer parallelism. The later parallelism combination has a separate 2021 milestone.

New constraints

Communication, bubbles and topology constrain different parallel layouts.

Primary sources and supported content

Primary source: method and results

• Megatron-LM · 2019-09-17 · Checked 2026-10-21

Relations

2021-04-09 · TP × PP × DP · Method

3D parallelism · TP × PP × DP

Bottleneck

One parallel dimension cannot efficiently scale very large models.

Mechanism

Combine tensor, pipeline and data parallelism around topology, microbatches, communication and bubbles.

Evidence and conditions

The 2021 work studies parallelism combinations for large-scale training; device count alone does not predict throughput.

New constraints

Communication, bubbles and topology constrain different parallel layouts.

Primary sources and supported content

Primary source: method and results

• Efficient Large-Scale Language Model Training on GPU Clusters · 2021-04-09 · Checked 2026-10-21

Relations

2023-09-25 · DeepSpeed Ulysses · Method

Sequence parallelism · DeepSpeed Ulysses

Bottleneck

Long-sequence activations exceed device capacity.

Mechanism

Use all-to-all to switch between sequence and head partitions, distributing attention computation.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Network topology and available heads constrain parallel scale.

Primary sources and supported content

Abstract: method and scope

• DeepSpeed Ulysses · 2023-09-25 · Checked 2026-10-22

Relations

• Cross-track support · RoPE

  Positional extensibility and distributed execution address different bottlenecks.

  • RoFormer: Enhanced Transformer with Rotary Position Embedding · 2021-04-20 · Checked 2026-10-21
  • DeepSpeed Ulysses · 2023-09-25 · Checked 2026-10-22

2026-10-13 · Gemini 4 Argon · Distributed training on TPUs · Model practice

Distributed training on TPUs · Gemini 4 Argon

Bottleneck

Sparse multimodal models require training at scale.

Mechanism

The model card identifies TPU, JAX and ML Pathways as the training stack.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Parallelism ratios and serving scheduling are not established by this disclosure.

Primary sources and supported content

Implementation: Hardware / Software

• Gemini 4 Argon Model Card · 2026-10-13 · Checked 2026-10-23

Relations

2022-05-27 · FlashAttention · Method

IO-aware attention · FlashAttention

Bottleneck

HBM traffic for attention matrices becomes the bottleneck.

Mechanism

Tiling and online softmax reduce HBM intermediates while preserving mathematical attention.

Evidence and conditions

FlashAttention in 2022 studies IO complexity and kernels. Less intermediate storage does not make dense interactions linear.

New constraints

Dense attention does not become linear-complexity attention.

Primary sources and supported content

Primary source: method and results

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · 2022-05-27 · Checked 2026-10-21

Relations

• Improvement · FlashAttention-2

  Improve parallelism and non-matmul overhead over IO-aware tiling.

  • FlashAttention-2 · 2023-07-17 · Checked 2026-10-22

2023-07-17 · FlashAttention-2 · Method

Attention parallelism · FlashAttention-2

Bottleneck

After reducing HBM traffic, work partitioning still limits occupancy.

Mechanism

Reduce non-matmul operations and reorganize work across thread blocks and warps.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Gains depend on sequence, head dimension and hardware; complexity is unchanged.

Primary sources and supported content

Abstract: method and scope

• FlashAttention-2 · 2023-07-17 · Checked 2026-10-22

Relations

• Improvement · FlashAttention

  Improve parallelism and non-matmul overhead over IO-aware tiling.

  • FlashAttention-2 · 2023-07-17 · Checked 2026-10-22
• Improvement · FlashAttention-3

  Reorganize asynchronous pipelines for Hopper.

  • FlashAttention-3 · 2024-07-11 · Checked 2026-10-22

2024-07-11 · FlashAttention-3 · Method

Hopper pipelines · FlashAttention-3

Bottleneck

Generic execution underuses Hopper asynchronous hardware.

Mechanism

Warp specialization and asynchronous pipelines overlap matrix work with transfers.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Hopper optimization does not establish gains on every GPU.

Primary sources and supported content

Abstract: method and scope

• FlashAttention-3 · 2024-07-11 · Checked 2026-10-22

Relations

• Improvement · FlashAttention-2

  Reorganize asynchronous pipelines for Hopper.

  • FlashAttention-3 · 2024-07-11 · Checked 2026-10-22
• Improvement · FlashAttention-4

  Retarget bottlenecks for asymmetric Blackwell scaling.

  • FlashAttention-4 · 2026-03-05 · Checked 2026-10-22

2025-01-02 · FlashInfer · Engineering

Inference kernels · FlashInfer

Bottleneck

Serving has heterogeneous KV layouts and request shapes.

Mechanism

Customizable attention kernels, JIT and load organization adapt to KV structures and serving stages.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Kernel selection, compilation and dynamic request management also cost resources.

Primary sources and supported content

Abstract: method and scope

• FlashInfer · 2025-01-02 · Checked 2026-10-22

Relations

2026-03-05 · FlashAttention-4 · Method

Blackwell kernels · FlashAttention-4

Bottleneck

Blackwell matrix throughput grows faster than exponential units and shared-memory bandwidth.

Mechanism

Rework asynchronous MMA pipelines; software exponentials and conditional softmax rescaling reduce non-matmul bottlenecks, with TMEM in backward.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

This targets hardware imbalance; comparisons must specify B200, precision and shapes.

Primary sources and supported content

Abstract: method and scope

• FlashAttention-4 · 2026-03-05 · Checked 2026-10-22

Relations

• Improvement · FlashAttention-3

  Retarget bottlenecks for asymmetric Blackwell scaling.

  • FlashAttention-4 · 2026-03-05 · Checked 2026-10-22

2023-09-12 · PagedAttention / vLLM · Method

Paged KV cache · vLLM

Bottleneck

Contiguous KV reservations cause fragmentation and over-reservation.

Mechanism

Map logical tokens to noncontiguous KV blocks, allocating and sharing on demand.

Evidence and conditions

The 2023 PagedAttention/vLLM paper studies KV management; it reduces allocation waste rather than directly shrinking each token’s representation.

New constraints

Does not reduce mathematical KV payload; complements representation changes.

Primary sources and supported content

Primary source: method and results

• Efficient Memory Management for Large Language Model Serving with PagedAttention · 2023-09-12 · Checked 2026-10-21

Relations

• Cross-track support · GQA

  Representation compression and memory paging are complementary.

  • GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
  • Efficient Memory Management for Large Language Model Serving with PagedAttention · 2023-09-12 · Checked 2026-10-21

2023-12-12 · SGLang · RadixAttention · Method

Prefix reuse · SGLang

Bottleneck

Multi-turn and branching requests recompute shared prefixes.

Mechanism

A radix tree organizes prefix KV reuse and eviction alongside scheduling.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Hit rate depends on workload; cache occupancy and tenant isolation still need management.

Primary sources and supported content

Abstract: method and scope

• SGLang / RadixAttention · 2023-12-12 · Checked 2026-10-22

Relations

2022-07 · Orca · Method

Iteration scheduling · Orca

Bottleneck

Request-level batching waits for slow requests and delays slot reuse.

Mechanism

Admit new requests and remove completed ones at iteration boundaries instead of waiting for a static batch.

Evidence and conditions

Orca at OSDI 2022 presents iteration-level scheduling and related designs; assess throughput with first-token and tail latency.

New constraints

High throughput does not equal good interaction. Report TTFT, inter-token latency, tail percentiles and SLO-compliant throughput alongside arrival rate and length distributions. Aggressive batching may increase throughput while slowing every user.

Primary sources and supported content

Primary source: method and results

• Orca: A Distributed Serving System for Transformer-Based Generative Models · 2022-07 · Checked 2026-10-21

Relations

• Improvement · Sarathi-Serve

  Iteration scheduling additionally controls prefill interference with decode.

  • Sarathi-Serve · 2024-03-04 · Checked 2026-10-22

2024-03-04 · Sarathi-Serve · Method

Chunked prefill · Sarathi-Serve

Bottleneck

Large prefills stall decode, coupling throughput and latency.

Mechanism

Chunk prefills and mix them with decode for more predictable iteration work.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Chunk size affects first-token and inter-token latency and must follow SLOs.

Primary sources and supported content

Abstract: method and scope

• Sarathi-Serve · 2024-03-04 · Checked 2026-10-22

Relations

• Improvement · Orca

  Iteration scheduling additionally controls prefill interference with decode.

  • Sarathi-Serve · 2024-03-04 · Checked 2026-10-22

2024-01-17 · DistServe · Engineering

Prefill/decode split · DistServe

Bottleneck

Prefill and decode need different resources and interfere.

Mechanism

Separate prefill/decode resources and transfer KV to optimize first-token and generation latency independently.

Evidence and conditions

DistServe in 2024 studies disaggregated goodput, conditioned on state transfer, resource ratios and SLOs.

New constraints

Transfers and resource imbalance can outweigh reduced interference.

Primary sources and supported content

Primary source: method and results

• DistServe: Disaggregating Prefill and Decoding · 2024-01-17 · Checked 2026-10-21

Relations

• Combination · Dynamo · PD

  Combine PD disaggregation with routing, KV transfer and resource planning.

  • Introducing NVIDIA Dynamo · 2025-03-18 · Checked 2026-10-21

2025-03-18 · Dynamo · PD · Engineering

Distributed PD serving · NVIDIA Dynamo

Bottleneck

PD deployment also requires routing, transfer and dynamic planning.

Mechanism

Dynamo introduces PD disaggregation and KV-aware routing across resource pools.

Evidence and conditions

NVIDIA discloses the design on March 18, 2025. Framework support does not mean every deployment enables it.

New constraints

Transfers and resource imbalance can outweigh reduced interference.

Primary sources and supported content

Primary source: method and results

• Introducing NVIDIA Dynamo · 2025-03-18 · Checked 2026-10-21

Relations

• Combination · DistServe

  Combine PD disaggregation with routing, KV transfer and resource planning.

  • Introducing NVIDIA Dynamo · 2025-03-18 · Checked 2026-10-21
• Improvement · Dynamo · EPD

  Multimodal serving separates encoding and specifies suitable workloads.

  • When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21

2026-09-09 · Dynamo · EPD · Engineering

Encode/prefill/decode · Dynamo · EPD

Bottleneck

Multimodal encoding and language execution have different resource needs.

Mechanism

Also separate multimodal encoding, independently scaling encode, prefill and decode and transferring state.

Evidence and conditions

The September 9, 2026 analysis discusses EPD conditions. Evaluate modality, input lengths, hardware, network and load.

New constraints

Transfers and resource imbalance can outweigh reduced interference.

Primary sources and supported content

Primary source: method and results

• When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21

Relations

• Improvement · Dynamo · PD

  Multimodal serving separates encoding and specifies suitable workloads.

  • When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21
• Cross-track support · Qwen3.5

  Multimodal models introduce encoding load; separation needs measurement and does not imply Qwen uses Dynamo.

  • Qwen3.5 · 2026-02-15 · Checked 2026-10-22
  • When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21

2024-12-27 · DeepSeek-V3 · DualPipe / FP8 · Engineering

Overlap and FP8 · DeepSeek-V3

Bottleneck

MoE communication lies on the compute critical path.

Mechanism

Bidirectional pipelines overlap communication and compute; FP8 with higher-precision accumulation and sensitive operations controls training cost.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Topology, scheduling and numerics must align; not all tensors use FP8.

Primary sources and supported content

Infrastructure / FP8 training

• DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21

Source revised 2025-02-18

Relations

• Cross-track support · DeepSeek-V3 · load balancing

  MoE training needs coordinated compute and expert communication.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
• Combination · DeepEP

  Pipeline scheduling and expert communication kernels optimize different layers.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
  • DeepEP: expert-parallel communication · 2025 · Checked 2026-10-22

2025 · DeepEP · Engineering

Expert communication · DeepSeek · DeepEP

Bottleneck

Expert dispatch and combine communication limit MoE efficiency.

Mechanism

Provide high-throughput and low-latency MoE communication kernels for training/prefill and decode.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Supported GPUs and networks are required; release timing does not redefine MoE origins.

Primary sources and supported content

README: kernels / requirements

• DeepEP: expert-parallel communication · 2025 · Checked 2026-10-22

Relations

• Combination · DeepSeek-V3 · DualPipe / FP8

  Pipeline scheduling and expert communication kernels optimize different layers.

  • DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
  • DeepEP: expert-parallel communication · 2025 · Checked 2026-10-22

2026-04-14 · Nemotron 3 Super · NVFP4 · Model practice

Low-precision training · Nemotron · NVFP4

Bottleneck

Lower-precision training must control both error and throughput.

Mechanism

The report uses NVFP4 pretraining, evaluating low precision within training rather than only quantizing deployment weights.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Matching hardware and scaling are needed; results do not apply to arbitrary 4-bit formats.

Primary sources and supported content

Abstract: method and scope

• Nemotron 3 Super · 2026-04-14 · Checked 2026-10-22

Relations

• Cross-track support · Nemotron 3 Super

  Architecture and low-precision pretraining in the same model.

  • Nemotron 3 Super · 2026-04-14 · Checked 2026-10-22

2024-09-28 · HybridFlow / verl · Method

Training/rollout loop · HybridFlow / verl

Bottleneck

RL has multiple model roles and different sharding for training and generation.

Mechanism

Hybrid control expresses distributed dataflow and reorganizes actor sharding between training and generation.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Placement, resharding and role scheduling depend on cluster and workload.

Primary sources and supported content

Abstract: method and scope

• HybridFlow · 2024-09-28 · Checked 2026-10-22

Relations

• Cross-track support · DAPO

  The DAPO implementation is built on verl.

  • DAPO · 2025-03-18 · Checked 2026-10-22
• Alternative · AReaL

  Multi-role orchestration and asynchronous rollout emphasize different system choices, not direct lineage.

  • HybridFlow · 2024-09-28 · Checked 2026-10-22
  • AReaL · 2025-05-30 · Checked 2026-10-22
• Alternative · Mistral Large 4 · Asynchronous long-trajectory RL

  An implementation comparison for training/rollout coordination; not a claim that Mistral uses HybridFlow.

  • Introducing Mistral Large 4 · 2026-10-06 · Checked 2026-10-23

2025-05-30 · AReaL · Method

Asynchronous RL · AReaL

Bottleneck

Synchronous RL waits for the longest output, leaving training devices idle.

Mechanism

Decouple continuous rollouts and training, controlling stale-policy samples through load balancing and staleness-aware updates.

Evidence and conditions

The primary source supports this mechanism; performance depends on its tasks, models and hardware, not a cross-setting benchmark by this site.

New constraints

Higher throughput brings staleness constraints; old trajectories cannot accumulate without bound.

policy lag = current update − rollout policy version

Primary sources and supported content

Abstract: method and scope

• AReaL · 2025-05-30 · Checked 2026-10-22

Source revised 2026-03-02

Relations

• Alternative · HybridFlow / verl

  Multi-role orchestration and asynchronous rollout emphasize different system choices, not direct lineage.

  • HybridFlow · 2024-09-28 · Checked 2026-10-22
  • AReaL · 2025-05-30 · Checked 2026-10-22
• Cross-track support · Kimi k1.5

  Long-output RL motivates asynchronous execution; this does not assert k1.5 uses AReaL.

  • Kimi k1.5 · 2025-01-22 · Checked 2026-10-22
  • AReaL · 2025-05-30 · Checked 2026-10-22

2026-10-06 · Mistral Large 4 · Asynchronous long-trajectory RL · Model practice

Asynchronous long-trajectory RL · Mistral Large 4

Bottleneck

Long tool trajectories make rollout latency expensive for training.

Mechanism

An autoscaling actor fleet generates rollouts while training proceeds asynchronously; tasks combine tests, reward models and checks.

Evidence and conditions

Disclosed in the cited section; not independently reproduced. This application date is not a claim of method invention.

New constraints

Policy staleness and off-policy drift must be controlled; throughput depends on configuration.

Primary sources and supported content

Reinforcement learning at scale

• Introducing Mistral Large 4 · 2026-10-06 · Checked 2026-10-23

Relations

• Alternative · HybridFlow / verl

  An implementation comparison for training/rollout coordination; not a claim that Mistral uses HybridFlow.

  • Introducing Mistral Large 4 · 2026-10-06 · Checked 2026-10-23