How do verifiable rewards shape reasoning?

The bottleneck

Supervised reasoning is limited by available high-quality traces. Some math answers and code outputs can be checked, providing RL signals through sampling and updates. The reward only covers what the verifier can actually check.

The mechanism

R1-Zero explores direct RL without preliminary SFT. R1 uses cold-start data and multiple stages. GRPO estimates advantages from relative rewards within a group of candidates for one prompt, then updates the constrained policy. Outcome rewards do not prove every reasoning step correct.

Âᵢ ≈ (rᵢ − mean(r_group)) / std(r_group)

Examples and evidence

Example: rewards [0,0,1,1] distinguish correct from incorrect candidates. Equal group rewards provide no such discrimination, and implementations must handle zero variance. OpenAI describes reasoning RL for o1, but this does not establish identical rewards or algorithms.

New constraints

Verifiers can be exploited and tests can be incomplete. Easily checked tasks do not represent all open-ended problems. Check generalization, readability, length costs and reward hacking; longer explanations alone are not evidence of better reasoning.