Nathan Lambert
Executive Director, Trillium Labs
Cornell University, November 20, 2026
Slides: natolambert.com/cornell-2026
o1 AIME accuracy vs. train-time and test-time compute (OpenAI, Sep. 2024)
Post-training is a more complete set of techniques and best practices to make base, next-token language models more useful for downstream tasks.
Three optimization methods: Supervised Finetuning (SFT), preference finetuning / human feedback / RLHF, and Reinforcement Learning with Verifiable Rewards (RLVR).
Modern post-training with many rounds (RLHF Book, ch. 3)
Post-training is a more complete set of techniques and best practices to make base, next-token language models more useful for downstream tasks.
Three optimization methods: Supervised Finetuning (SFT), preference finetuning / human feedback / RLHF, and Reinforcement Learning with Verifiable Rewards (RLVR).
The term originated at OpenAI and was first used publicly in a prominent fashion with the release of GPT-4 in 2023:
“GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior.”
The shape of a post-training recipe has changed more in the last year than in the prior three.
The recipe:
Stages: Stage 1 SFT/reasoning foundation → Stage 2 train domain-specialist teachers (with older-style post-training recipes) → Stage 3 MOPD into a single student (final model).
Lineage: MiMo-V2-Flash introduced MOPD → DeepSeek V4, Nemotron 3 Ultra, and others use it as well (e.g. latest GPT models).
Specialist post-training with MOPD (RLHF Book, ch. 3) (Xiaomi LLM-Core Team et al., 2026)
Share of Anthropic + OpenAI compute by workload, 1Q24 to 4Q26 (est.). SemiAnalysis, via OfficeChai
RL compute relative to pretraining compute for the same model, in GPU-hours:
| Model | Pretraining | RL | RL / pretraining |
|---|---|---|---|
| DeepSeek R1 Jan 2025 671B total, 37B active |
2.66M H800 GPU-hours (DeepSeek-AI, 2025) | 0.14M H800 GPU-hours (Guo et al., 2025) | ~5% |
| Beam Oct 2026 501B total, 23B active |
< 4.1M 6,144 GB300s for under 4 weeks | ~7.1M 10.5K GB300s for 4 weeks | > 1.7× |
| Inkling 2 Small Nov 2026 594B total, 17B active |
~0.11M* ~1.5K GPUs for under 3 days | ~0.44M 2,048 GPUs for 9 days | ~4×* |
Many companies and researchers start with the final reasoning model, rather than the base model.
Post-training: Building the core reasoning behavior of models. Now very expensive (O($50M+/yr) to do it well).
Post-post-training: Adapting a post-trained model to specific domains, efficiencies, capabilities, harnesses, etc.
As of Nov. 19, 2026. Raw data (CSV)
Harvey, Aug. 2026: a Kimi K3 base post-trained with Fireworks for legal work, so law firms can own their models
Mercor and SkyRL, Sep. 2026: Qwen3.5-397B-A17B post-trained on expert data, with the script and weights released
Harvey, Aug. 2026: a Kimi K3 base post-trained with Fireworks for legal work, so law firms can own their models
Harvey, Aug. 2026: quality-cost frontier on Legal Agent Bench hold-out tasks; Tenet nearly doubles the all-pass rate of its Kimi K3 base at a stable cost
TMax: an open RL recipe that turns released open-weight Qwen models, 2B to 27B, into terminal agents (Ivison et al., 2026).
Context Language Models (CLM): existing models, zero-shot or with online RL (Qwen3.5-9B), manage their own context directly rather than through structured harness commands (Shao et al., 2026).
Ranked from least to most compute intensive.
+ unlocking new domains, cross-disciplinary post-training, and applications with low-revenue prospects (labs will ignore)
The design space for how we wrangle models into productive work on specific tasks, especially with specialization & memory, is very underexplored.
What are the main abstractions and methods for doing this work?
+ unlocking new domains, cross-disciplinary post-training, and applications with low-revenue prospects (labs will ignore)
Example: CLMs allow the models to edit their context directly (rather than relying on compaction, harness commands, etc.), which enables fascinating RL specialization options & KV cache management.
+ how models get their values & personality, how it translates to agentic actions, and trade-offs between methods like Constitutional AI vs. Model Specs
+ how models get their values & personality, how it translates to agentic actions, and trade-offs between methods like Constitutional AI vs. Model Specs
In our work studying this training, Open Character Training, measuring performance is extremely challenging.
The field in industry has been largely rebuilt, with OpenAI releasing a new Model Spec for agents – open literature has not kept up.
(This is one of our first areas at Trillium Labs).
Traits whose revealed preference (Elo) changed most after character training, Llama 3.1 8B (Maiya et al., 2025)
+ synthetic data methods for RL, RL env cleaning & filtering
Post-training research does not have the data commons to seed it like pretraining did!
It will take major individual & institutional efforts to fix this. Nemotron, Olmo, OpenThoughts, etc. are a good start, but far from enough given the importance.
+ synthetic data methods for RL, RL env cleaning & filtering
A large bottleneck on making good data is the obsessiveness to deeply understand how models interact with your data.
It’s a common occurrence for me to hear how a good data or eval project has ended with the lead author re-reading every single datapoint.
Still, academics regularly create datasets which could be worth meaningful revenue on the frothy SF data market.
TMax, a project to create terminal-bench data and post-post-train small Qwen models to SOTA numbers.
+ manipulating CoT efficiency, and alignment
Why are there no papers that are clearly showing what we need to do in post-training to make massive token efficiency gains like the frontier labs? We can do this at smaller scales, on simpler problems to establish the science.
Agentic coding score vs. total tokens at each effort level. Anthropic, Claude Opus 4.7, Apr. 2026
+ manipulating CoT efficiency, and alignment
A simple idea (partially related): What if we spend more RL compute on the hardest problems?
Recall: The popular algorithm Group Relative Policy Optimization (GRPO) generates multiple completions to the same prompts to try and create batches that have correct & incorrect answers (those have gradient).
Can we create batches dynamically around this to better allocate compute?
Motivation: Common RL practices improve performance on easy problems, but not the hardest problems.
A proposed solution: Never Give Up (NGU) – expand inference batches if all completions are incorrect, with probability p.
This is one of my papers where I’ve been the most confident in the direction of the idea, but it’s very difficult to make the RL experiments fair! Shockingly hard!
+ designing methods like pretraining ladders for post-training, and RL data selection
We need to design methods for predicting performance of large post-training runs. What will the equivalent of a pretraining ladder look like for RL?
+ evaluation of swarms, and formulations for how models collaborate reliably & efficiently
Multi-agent systems are a way to spend more on compute to:
a) make LLMs faster in wall-clock time, and
b) increase peak performance of the models on hard tasks.
+ evaluation of swarms, and formulations for how models collaborate reliably & efficiently
SEC-Bench Pro score vs. latency with 1, 4, and 16 agents. OpenAI, GPT-5.6, Jul. 2026
+ evaluation of swarms, and formulations for how models collaborate reliably & efficiently
Lean theorem proving: best score within 2, 6, and 24 hours, by team size. Anthropic, Claude Opus 5.5 system card, Sep. 2026
RSI: Recursive Self-Improvement – using AI models to increasingly automate and accelerate AI research / model building.
I constantly hear variants of:
Share of Anthropic's model R&D tasks at each automation level, Aug. 2025 to Aug. 2026. Anthropic, Nov. 2026
Is this measured by time spent?
Is this measured by percent of code written or experiments run?
To use a Claude-ism, the load-bearing piece of work may be a very small proportion of the overall process.
Researchers are trained for and valued for that small, insightful piece of work.
Share of Anthropic's model R&D tasks at each automation level, Aug. 2025 to Aug. 2026. Anthropic, Nov. 2026
More of science shifts to problem formulation, understanding “impact”, communication, standard setting, consensus building, mentorship, etc.
For senior scientists: I barely write code anymore, but I’m busier than ever as a scientist. LLMs are clearly very empowering.
For junior scientists: Doing inefficient work manually is often the only way to build understanding. Training could take longer, with more work feeling like “homework” rather than productive output.
In 2023, it took a long time for clear, almost obvious, ideas to be used in strong models! In this era, the goal was to match performance of closed models on “chat” evals with minimal finetuning.
MT-Bench by category (Tunstall et al., 2024)
RewardBench: Building the first evaluation for reward models used in RLHF.
Would expect this to take ~1/2 to 1/4 of the time now.
Olmo: In the first Olmo, the post-training was incredibly minimal, mostly the Tülu 2 recipe, added almost as an afterthought. Early in 2024, building a language model was way more about pretraining… can figure out how to “align” it later.
Models can do this “apply known solution” work fairly well.
Olmo 3 model flow (Olmo et al., 2025)
The goals we have in building new research infrastructure & recipes:
Slides: natolambert.com/cornell-2026
Newsletter: interconnects.ai
Book: rlhfbook.com
Trillium Labs: trilliumlabs.org
We’re hiring: full-time MTS, student researchers, and interns
Contact: nathan@natolambert.com
Estimated daily AI R&D tasks per researcher at OpenAI by type of work, Mar. to Oct. 2026. OpenAI, Nov. 2026
Pretraining a model and then finetuning it was standard practice in computer vision before it took over NLP.