Where should extra inference compute go?

The bottleneck

With parameters fixed, answering compute can still vary. Hard problems may benefit from longer reasoning or candidate search; simple problems may not justify the same budget. Research in 2024 connected allocation to difficulty, while products exposed thinking budgets.

The mechanism

Serial compute extends one trajectory; parallel compute produces candidates; a verifier or ranker selects among them. Account for generation and scoring, not just returned tokens. Difficulty, model and verifier quality change the best allocation.

C_total = Σ C_generate(candidateᵢ) + C_verify + C_select

Examples and evidence

Eight short candidates and one long trajectory may have similar token budgets but different critical-path latency and correlated errors. o1, Claude 3.7 and Qwen3 disclose different reasoning/thinking capabilities, without establishing identical internal search.

New constraints

More compute can reinforce mistakes. An unreliable verifier may select candidates that exploit scoring. Measure task-specific quality–cost–latency curves and define stopping rules; longer thinking is not universally better.