The Balance of Open Models

Nathan Lambert

Executive Director, Trillium Labs & Founder, Interconnects

Takeoff 2026, Washington, DC, November 14, 2026

Lambert 1/33

Open model inference exponential growth

Open model inference exponential growth

Lambert 2/33

Open model inference growth (compared to Google!)

Open model inference growth (compared to Google!)

Lambert 3/33

Open models are catching up as the foundation of R&D

Qwen is mentioned in almost as many papers as OpenAI's GPT models

Lambert 4/33

Open weights trail the frontier by a few months

Artificial Analysis Progress Index v1 frontier over time, open weights vs. proprietary: Kimi K3.1 Pro is about 2 months behind and DeepSeek V4.1 Flash about 4 months behind

Artificial Analysis Progress Index v1, October 30, 2026. Kimi K3.1 Pro counted as open; AA lists it as proprietary until its weights ship.

Lambert 5/33

How are open models built?
The role of distillation

Lambert 6/33

Training a model in three stages

Pre-training
→
Mid-training / SFT
→
RL
  • Large-scale web text
  • Creates the base model: the foundation of knowledge
  • Highest-quality web text, plus reasoning traces that start teaching chain-of-thought behavior; crucial to scaling RL well
  • Strongly overlaps with SFT (supervised finetuning), the first post-training stage, trained on full reasoning traces to prepare for RL
  • Unclear whether a "true base model" with minimal CoT ability still exists, publicly or inside labs
  • This is the primary location for distillation.
  • Agentic RL across ~1M environments; big runs use 10K+ GPUs for weeks at a time
  • On balance RL now uses more compute than pre-training, much of it inference for rollouts: Beam pre-trained on 6,144 GB300s for under four weeks, then ran RL on 10.5K GB300s for four weeks with 100M+ rollouts
Lambert 7/33

Define: Distillation

Distillation generally comes from the technical line of work on teacher-student knowledge distillation, a loss function that learns the shape of a distribution rather than just if the next token prediction was correct (Hinton et al., 2015).

Distillation in post-training became to be known as the set of tools for using the outputs of any model to train a downstream model (ie. most synthetic data) (Tunstall et al., 2023).

Distillation today refers to using reasoning traces from the latest models (which is hidden in APIs) to train a competitive model. A better term for this is adversarial distillation or jailbreaking reasoning APIs as it is more about illicit access than the above technical methods.

More on the history of distillation is here in my book.

Lambert 8/33

Wait, I thought reasoning traces were hidden?

Very common among Chinese research labs to have teams that work to get raw reasoning tokens back from API models.

The core of the distillation story is accessing full reasoning traces that are hidden from users.

arXiv:2608.09867

(Panfilov et al., 2026)

Lambert 9/33

Why do the labs think distillation is such a big deal?

  • Scale: Anthropic has detailed very wide-scale usage of their models by Chinese AI labs, which has captured sensitive Chinese government data getting routed to their services (Anthropic, 2026)
  • Sophistication: Many actors in the Chinese AI ecosystem have dedicated teams to circumvent the intended use of the API, extracting reasoning tokens (e.g. methods from the paper on previous slides)
  • Private / classified briefings: As with cyber risks (more on this later), there are regular classified briefings on AI matters relevant to national security
Lambert 10/33

Why do I think distillation is not worth focusing on?

  • Unclear impact on performance: The best public research (open-thoughts, olmo, etc.) on using reasoning traces in post-training does not show this as a trivial pipeline in SFT to massive gains. Much public work is indicating an increasing role of RL today.
  • … what policy levers? There is not good short-term action that can be taken against Chinese labs who are distilling without massively setting back the open ecosystem globally
  • APIs have been insecure to distillation: The labs are only starting to secure their APIs against distillation
  • The Chinese labs are very good at building models – if distillation is banned they will still be close competitors. Fast-following is much more compute efficient than pushing the frontier.
  • Usage by Chinese labs isn’t all for training: It’s unclear if a lot of the volume Chinese labs access is used for training. This would rapidly diminish urgency.

Generally: Very little public information/research on the impact of distillation, few government actions that seem helpful, good cases that it should be legal (competition, especially to help the US open model economy).

Lambert 11/33

Accessing banned models in China

Claude is banned in China, but developers still love it.

  • Supply: farmed free credits, resold quotas, split Max plans, stolen cards; one network ran 20,000+ fraudulent accounts (Anthropic, Feb 2026)
  • Tokens are often subsidized to capture usage data and resell later
  • Why do Chinese labs route requests on their products to Claude? Maybe improve downtime, maybe collect a different distribution of user data for analysis, … doesn’t fit into traditional “distillation for training” idea-space (I’m very confused by this)
ChinaTalk: How to Buy Cheap Claude Tokens in China, The Transfer Station Economy, Explained

Zilan Qian, ChinaTalk, May 5, 2026

Lambert 12/33

Open models occupy a slim slice of the price Pareto frontier

Artificial Analysis: Intelligence Index vs. cost per task

Artificial Analysis, Intelligence Index vs. cost per task, October 2026

Lambert 13/33

Mapping the open model economy

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee · others
Lambert 14/33

Revenue growth is key to continued training-scaling, and often captured elsewhere

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
Serve and post-train
Fireworks · Baseten · Together
Lambert 15/33

Competing with integration and vast scale

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
Serve and post-train
Fireworks · Baseten · Together
OpenAI and Anthropic own the whole stack
Products and integrations
Specialization tools
Models
Compute
Chips and data centers
Lambert 16/33

The opening of an opportunity: enterprises want to own their IP and context

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
Serve and post-train
Fireworks · Baseten · Together
Specialize and hillclimb
Prime Intellect · Applied Compute · Mercor
OpenAI and Anthropic own the whole stack
Products and integrations
Specialization tools
Models
Compute
Chips and data centers
Lambert 17/33

A near-term challenge – convergence of business models

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
Serve and post-train
Fireworks · Baseten · Together
Specialize and hillclimb
Prime Intellect · Applied Compute · Mercor
Post‑training tooling
Convergence of business models
Lambert 18/33

Model builders have the resources to compete in inference

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
↓ Pressure to move downstream
Serving tokens
Fireworks · Baseten · Together
Specialize and hillclimb
Prime Intellect · Applied Compute · Mercor
Post‑training tooling
Convergence of business models
Lambert 19/33

Open models are economically fragmented

Building models
Moonshot · Z.ai · DeepSeek · Alibaba · Reflection · Arcee
Serve and post-train
Fireworks · Baseten · Together
Specialize and hillclimb
Prime Intellect · Applied Compute · Mercor
OpenAI and Anthropic own the whole stack
Products and integrations
Specialization tools
Models
Compute
Chips and data centers
Lambert 20/33

The rising costs of training a truly frontier model are looming
Labs will start getting pushed out

Is the value in open models fully resting on keeping up?

Jensen Huang on X: GPT-6 Astra, trained on ~100K+ NVIDIA Grace Blackwell NVLink72. 400K GPUs coming online next.

Jensen Huang, September 6, 2026

Lambert 21/33

The task for open models – build the cheapest, most adaptable, private intelligence

Lambert 22/33

GLM 5.2 was a step change for open agents

  • Used by researchers across industry to complement Claude/Codex
  • Finetunable to specialized workflows
  • The first open model to have agentic product-market-fit like Opus 4.5 in December 2025
Interconnects: GLM-5.2 is the step change for open agents, Nathan Lambert, June 22, 2026

Interconnects, June 22, 2026

Lambert 23/33

What comes next with open models?

Lambert 24/33

Cyber risks of AI models have become real

Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign

Nov 13, 2025 · Anthropic: a Chinese state group ran 80–90% of an espionage campaign against ~30 targets through Claude Code

OALABS Research: Captured logs reveal hackers using Claude and Codex to breach companies

Jun 16, 2026 · OALABS: 1,000+ Claude Code and Codex sessions recovered from an attacker's staging server; 14+ companies breached

Anthropic: Detecting and countering misuse of AI, September 2026

Sep 10, 2026 · Anthropic: a Russian state-nexus group automated espionage against 20+ organizations with Claude; in a 1 TB data theft "AI agents performed nearly all of the work"

Hacktron: Hacking OpenAI

Sep 13, 2026 · Hacktron: Claude Opus wrote the exploit chain that compromised OpenAI employees' ChatGPT accounts and reached internal repos (disclosed research)

Lambert 25/33

Banning open models only kneecaps economic activity

A “ban” of open models would likely come from regulatory uncertainty over the use of certain models, especially those from China. In the case of a regulatory ban, bad actors trying to use the powerful open models will almost certainly still have access. The likelihood of the US instituting a “great firewall” against the import of open weight models effectively is 0.

At the same time, there are many labs making models within ~1 year of the frontier of progress (10+) and the cost is still in the hundreds of millions of dollars, creeping towards billions, so many sovereigns – governments or large companies – can still build and release these models.

Lambert 26/33

… risks of open-weight models have been exaggerated for years!

Many other risks have been discussed with open-weight models. Many of posited risks of open models turned out to be overblown – such as misinformation – in this case, the risks tend to involve complex forms of distribution and/or have existing human bottlenecks.

Some risks, e.g. non-consenual deepfakes, are clearly accelerated by open-weight models. Once these thresholds are passed, you can’t “un-release” the model, so the path forward requires new societal preparation.

arXiv:2403.07918

(Kapoor et al., 2024)

Lambert 27/33

How to manage marginal risks of open-weight models

The organizations at the front of open models should look at each model and consider the marginal risks produced by their models. Today, different AI labs have very different reads on types of risks (e.g. xAI vs Anthropic), so we need national standards on risk rather than relying on each company.

This Thinking Machines article is a great read on the balance of marginal risks alongside the benefits of openness.

thinkingmachines.ai

Thinking Machines, July 2026

Lambert 28/33

Open models are crucial to cyber defense; closed models are powerful vectors of cyber risk

Recent events have shown open models to be crucial for cyber defense (HuggingFace, OpenCode incidents), as trusted access programs run by companies cannot easily scale to the entire economy.

Closed models have been used in numerous cyber attacks, and there is no research showing that their safeguards are on track to become steadfast.

As it stands, the balance is more that both open and closed models are increasing risk, but bottlenecks exist and cyber is a complex process to diffuse. Erring on the side of helping the defender is my bet (but go read more experts on cyber).

Lambert 29/33

This is one of multiple issues/risks in AI’s diffusion that we will all face!

The research community and broader society relies on open models as a tool to understand and prevent said risks!

Lambert 30/33

Open models are at the core of long-term diffusion, reduction in concentration of power, safety-research, and many other things we need to make AI go well! We must, navigate these challenging issues along the way!

Lambert 31/33

Conclusions

Thanks for listening!

Slides: https://natolambert.com/slides/2026/takeoff-dc/talk.html

Newsletter: interconnects.ai

Data: dashboard.interconnects.ai/

Contact: @natolambert / nathan@natolambert.com

Lambert 32/33

References

Hinton, G., Vinyals, O., and Dean, J. “Distilling the Knowledge in a Neural Network.” 2015. [link]
Kapoor, S., Bommasani, R., Klyman, K., and others. “On the Societal Impact of Open Foundation Models.” 2024. [link]
Panfilov, A., Schmotz, D., Shumailov, I., and others. “Stealing Reasoning Traces from Proprietary LLM APIs.” 2026. [link]
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., et al. “Zephyr: Direct Distillation of LM Alignment.” 2023. [link]
Lambert 33/33

Extra slides

Lambert 34/33

Xiaomi streamed the MiMo-V2.6 RL runs live

The evidence that China isn’t solely reliant on distillation is plentiful.

  • MiMo-V2.6-Pro (1.0T, 42B active)
    • 30 RL steps in 5 days 7 h
    • 753k rollouts (1,568 prompts × 16 per step), 75B tokens
    • $2.62M
  • MiMo-V2.6-Flash (309B, 15B active)
    • 30 RL steps in 3 days 11 h
    • 753k rollouts, 81.4B tokens
    • $854k
  • Both runs: ~1.5M rollouts, $3.47M on the dashboard’s meter
Xiaomi MiMo RL dashboard: run stats, benchmark curves, and training metrics for the MiMo-V2.6 Pro and Flash runs

mimo.xiaomi.com/rl: public dashboard of the MiMo-V2.6 RL runs (Sep 15–20, 2026), captured Oct 20, 2026

Lambert 35/33

The lasting advantage: US frontier labs have roughly 10×+ the compute of leading Chinese labs

AI compute per lab, H100-equivalents, log scale. US, end-2025, median and 90% interval China, 2025–26

OpenAI
1.7M
Google DeepMind
1.6M
Anthropic
1.2M
Meta Superintelligence Labs
996k
xAI (SpaceXAI)
635k
Leading Chinese labsDeepSeek, Moonshot, Zhipu, Qwen…
100–200 MW ≈ 71k–167k, at most
DeepSeek, Jan 2025shared with High-Flyer
≤ 50k Hopper GPUs
20k50k100k200k500k1.0M2.0M4.0M

Epoch AI, AI Chip Users Explorer (Dec 31, 2025; research, training, and inference, incl. rented cloud) · Dylan Patel, Aug 2026: leading Chinese labs have 100–200 MW at most, converted at 1.2–1.4 kW per H100-equivalent · SemiAnalysis, Jan 2025: DeepSeek ~50k Hopper GPUs

Lambert 36/33

The lasting advantage: US frontier labs have roughly 10×+ the compute of leading Chinese labs

16×
average total-compute advantage of the top five US labs over the top five Chinese labs, end-2026 estimate
IFP, Oct 19, 2026, citing a forthcoming Pilz (2026) report
25–50×
more than 5 GW each at Anthropic and OpenAI by end of 2026, vs. 100–200 MW at most at a leading Chinese lab (ByteDance Seed aside)
Dylan Patel on the Dwarkesh Podcast, Aug 25, 2026
Lambert 37/33

Win condition 1: Driving the cost of proven LLM workflows to 0

Lambert 38/33

Win condition 1: Driving the cost of proven LLM workflows to 0

Detour / Challenge: Astronomical demand for compute is maintaining high margins across open & closed model providers.

Lambert 39/33

Win condition 1: Driving the cost of proven LLM workflows to 0

Detour / Challenge: Astronomical demand for compute is maintaining high margins across open & closed model providers.

Open models will only win when we’re willing to drive prices down to commodity pricing, slightly above the price of compute, or even below to gain market share.

Lambert 40/33

Win condition 2: Adaptable, private, controlled intelligence

Lambert 41/33

Win condition 2: Adaptable, private, controlled intelligence

Frontier labs are not going to be willing to deploy their models on-premise for companies who want to control their internal expertise, data, and IP in the face of super-powerful AI.

Lambert 42/33

Win condition 2: Adaptable, private, controlled intelligence

Frontier labs are not going to be willing to deploy their models on-premise for companies who want to control their internal expertise, data, and IP in the face of super-powerful AI.

Open models are the only viable path for companies that want to build their own agents and deep AI ownership.

Lambert 43/33