3 Paths Open-Source LLMs Use to Chase the Frontier: Distillation, MoE & Synthetic Data

TL;DR: DeepSeek V4 and Qwen3 are delivering GPT-4-level performance at one-tenth or less of the cost. Three technologies make this possible: distillation, MoE (Mixture of Experts) sparse architecture, and synthetic data. This deep dive analyzes how each works, its limitations, and what open-source LLMs catching up to the frontier means for the AI industry landscape.

---

Why did open-source LLMs suddenly get so capable?

Until 2024, open-source LLMs were clearly behind GPT-4. Then 2025–2026 changed that.

Open-source LLM benchmark snapshot, March 2026:

Model	Organization	Parameters	MMLU	Cost ($/1M tokens)
GPT-4o	OpenAI (commercial)	Undisclosed	88.7	$5/$15
Claude Sonnet 4.6	Anthropic (commercial)	Undisclosed	90+	$3/$15
DeepSeek V4	DeepSeek (open)	1T (active: 37B)	87.1	$0.27/$1.1
Qwen3-72B	Alibaba (open)	72B	86.9	$0.5/$1.5
Llama 3.3-70B	Meta (open)	70B	85.7	Free (self-hosted)

DeepSeek V4's API cost is approximately 18–20x cheaper than GPT-4o. Performance difference: 1–2%. How is this possible?

---

Path 1: What is knowledge distillation and how does it work?

The concept: teacher and student

Knowledge distillation is a technique proposed by Geoffrey Hinton's team in 2015. It transfers the knowledge of a large "teacher model" to a smaller "student model."

Traditional training approach:

Student model sees training data and learns correct labels
Cat image → correct label: cat

Distillation approach:

Teacher model predicts first: "cat 0.95, dog 0.03, rabbit 0.02"
Student model learns from these "soft labels"
Far more informational (probability distribution reveals more than just "cat")

The teacher model's output probability distribution contains relational information between data points. The student model learns more efficiently — absorbing that cats and dogs are similar, rabbits less so.

Distillation in modern LLMs

Modern LLM distillation is more sophisticated:

1. Response distillation: high-quality answers generated by GPT-4 or Claude used as training data for smaller models
2. Chain-of-Thought (CoT) distillation: smaller models learn the step-by-step reasoning process of larger models
3. Preference distillation: reward signals from a larger model's RLHF process are transferred to smaller models

Microsoft's Phi series is a flagship distillation example. Phi-2 (2.7B parameters) demonstrated GPT-3.5 (175B) level performance on specific tasks.

Distillation's limits

• Cannot exceed the teacher: a student can't learn what the teacher doesn't know
• Reduced creativity: compressing the teacher's patterns can diminish creative reasoning capability
• License issues: training on outputs from commercial models (GPT-4, Claude) may violate terms of service

---

Path 2: How does MoE architecture reduce costs?

The concept: a team of specialists

MoE's idea is "don't activate all parameters all the time."

Traditional dense model:

Input → [100% of all parameters activated] → Output

MoE model:

Input → [Router: decides which experts to use] → [Only 2–4 selected experts activated] → Output

Using DeepSeek V4 as an example:

• Total parameters: 1 trillion
• Parameters activated when processing each token: 37 billion (3.7%)

The model theoretically holds the knowledge of 1 trillion parameters, but inference uses only 37 billion. This is the core mechanism enabling GPT-4-level performance at dramatically lower inference cost.

How MoE works

The router determines, for each input token, "which expert is best suited to process this token?"

For example:

• "Write Python code" → activates the coding expert cluster
• "Explain a historical event" → activates the history/social sciences expert cluster
• "Solve a math problem" → activates the math/logic expert cluster

This specialization allows the model to operate efficiently across a wider range of domains.

DeepSeek V4's MoE innovations

DeepSeek V4 added two innovations to standard MoE:

1. DeepSeekMoE: increases the number of experts while making each one smaller, enabling finer-grained specialization
2. MLA (Multi-head Latent Attention): compresses the attention cache to reduce memory usage

These two innovations are the core of reducing training costs to roughly one-tenth of traditional large models while maintaining performance.

MoE's limits

• Load balancing problem: if load concentrates on certain experts, efficiency drops
• Deployment complexity: storing 1 trillion parameters still requires large amounts of VRAM (~600GB at 4-bit quantization)
• Training instability: router training is unstable, making large-scale training complex

---

Path 3: What is synthetic data and why does it matter?

Why is synthetic data needed?

The internet text data used to train frontier LLMs is increasingly exhausted. As of 2026, most high-quality internet text has already been used to train large models.

The solution is synthetic data: existing LLMs generate new training data.

Three types of synthetic data

1. Reasoning process synthesis (Chain-of-Thought synthesis)

Have GPT-5 solve math problems, generating step-by-step solution processes
→ Use these solution processes as training data for smaller models
→ Smaller models learn step-by-step reasoning ability

2. Conversation data synthesis

Use a large model to generate high-quality conversation data across diverse scenarios
→ More varied and consistent conversation patterns than simple chat data
→ Smaller model learns rich conversational ability

3. Domain-specific synthesis

When specialized data for specific domains (medical, legal, coding) is scarce,
generate synthetic data using large model + domain expert review

Synthetic data success stories

Microsoft's Phi-3 series is a flagship synthetic data success. By mass-generating "textbook-quality text" using GPT-4, a 3.8B parameter model outperformed Llama 2-70B.

Qwen3 also uses synthetic data aggressively — particularly step-by-step solution processes generated by GPT-4 in math and coding domains contributed significantly to performance gains.

Synthetic data's limits

• Model collapse: training exclusively on synthetic data reduces diversity and accumulates errors
• Bias amplification: the teacher model's biases can be passed to the student model intact or amplified
• Factual error propagation: incorrect information generated by the teacher model can enter the student model's training data

---

How the three paths combine: DeepSeek V4's recipe

DeepSeek V4 combined all three technologies:

1. MoE: 1 trillion total parameters, 37B activated during inference → 90% reduction in inference cost
2. Synthetic data: math and coding domain synthetic data to strengthen reasoning capability
3. Indirect distillation: training targets set to match GPT-4-level performance within open-source license scope

Result: API cost 18x cheaper than GPT-4o, performance difference 1–2%.

---

What does open-source LLMs' catch-up mean for the AI industry?

Why is model commoditization accelerating?

The closer open-source LLMs get to frontier performance, the less differentiation value LLMs themselves carry. This is the "model commoditization" phenomenon we discussed.

OpenAI and Anthropic are shifting their competitive axis from model performance to ecosystem, API convenience, and enterprise trust.

What advantages does self-deploying open-source LLMs give enterprises?

When enterprises deploy open-source models like DeepSeek V4 or Qwen3 on their own servers:

• Data doesn't leave for external APIs (security)
• No API costs (economics)
• Can fine-tune on proprietary data (customization)

These three factors are accelerating enterprises' move to build their own AI infrastructure.

The geopolitical context

DeepSeek V4 and Qwen3 are open-source models made by Chinese companies. The US government has expressed national security concerns about these models. Some US government agencies and defense contractors restrict using Chinese open-source models.

Enterprises should make model selection decisions with awareness of this geopolitical context.

---

Key action summary

Technology	Principle	Representative examples	Limits
Knowledge distillation	Transfer knowledge from large model to small	Phi series, Qwen3	Cannot exceed teacher
MoE	Activate only a fraction of total parameters	DeepSeek V4, Mixtral	Deployment complexity, VRAM
Synthetic data	AI generates AI training data	Phi-3, Qwen3 coding	Model collapse, bias amplification

---

FAQ

Further reading

• This Week in AI: After NVIDIA GTC — Vera Rubin, Agent Runtime & Physical AI
• Year One of Physical AI: The 2026 Signals of Robots, Autonomous Vehicles & Drones Running on LLMs
• [Road to AI 08] The Transformer Revolution: Attention Is All You Need

Update notes

• First published: 2026-03-28
• Data basis: DeepSeek V4 technical report (January 2026), Qwen3 technical documentation (March 2026), Artificial Analysis benchmarks (March 2026)
• Next update: When major open-source model performance updates or new MoE architecture research is published

References

• DeepSeek V4 Technical Report (arXiv)
• BentoML: Best Open-Source LLMs 2026
• Hinton et al.: Distilling the Knowledge in a Neural Network
• Artificial Analysis: LLM Intelligence Index
• Ollama: Run LLMs Locally