TL;DR: DeepSeek V4 and Qwen3 are delivering GPT-4-level performance at one-tenth or less of the cost. Three technologies make this possible: distillation, MoE (Mixture of Experts) sparse architecture, and synthetic data. This deep dive analyzes how each works, its limitations, and what open-source LLMs catching up to the frontier means for the AI industry landscape.

Why did open-source LLMs suddenly get so capable?

Until 2024, open-source LLMs were clearly behind GPT-4. Then 2025–2026 changed that.

Open-source LLM benchmark snapshot, March 2026:

Model	Organization	Parameters	MMLU	Cost ($/1M tokens)
GPT-4o	OpenAI (commercial)	Undisclosed	88.7	$5/$15
Claude Sonnet 4.6	Anthropic (commercial)	Undisclosed	90+	$3/$15
DeepSeek V4	DeepSeek (open)	1T (active: 37B)	87.1	$0.27/$1.1
Qwen3-72B	Alibaba (open)	72B	86.9	$0.5/$1.5
Llama 3.3-70B	Meta (open)	70B	85.7	Free (self-hosted)

DeepSeek V4's API cost is approximately 18–20x cheaper than GPT-4o. Performance difference: 1–2%. How is this possible?

Path 1: What is knowledge distillation and how does it work?

The concept: teacher and student

Knowledge distillation is a technique proposed by Geoffrey Hinton's team in 2015. It transfers the knowledge of a large "teacher model" to a smaller "student model."

Traditional training approach:

Student model sees training data and learns correct labels
Cat image → correct label: cat

Distillation approach:

Teacher model predicts first: "cat 0.95, dog 0.03, rabbit 0.02"
Student model learns from these "soft labels"
Far more informational (probability distribution reveals more than just "cat")

The teacher model's output probability distribution contains relational information between data points. The student model learns more efficiently — absorbing that cats and dogs are similar, rabbits less so.

Distillation in modern LLMs

Modern LLM distillation is more sophisticated:

Response distillation: high-quality answers generated by GPT-4 or Claude used as training data for smaller models
Chain-of-Thought (CoT) distillation: smaller models learn the step-by-step reasoning process of larger models
Preference distillation: reward signals from a larger model's RLHF process are transferred to smaller models

Microsoft's Phi series is a flagship distillation example. Phi-2 (2.7B parameters) demonstrated GPT-3.5 (175B) level performance on specific tasks.

Distillation's limits

Cannot exceed the teacher: a student can't learn what the teacher doesn't know
Reduced creativity: compressing the teacher's patterns can diminish creative reasoning capability
License issues: training on outputs from commercial models (GPT-4, Claude) may violate terms of service

Path 2: How does MoE architecture reduce costs?

The concept: a team of specialists

MoE's idea is "don't activate all parameters all the time."

Traditional dense model:

Input → [100% of all parameters activated] → Output

MoE model:

Input → [Router: decides which experts to use] → [Only 2–4 selected experts activated] → Output

Using DeepSeek V4 as an example:

Total parameters: 1 trillion
Parameters activated when processing each token: 37 billion (3.7%)

The model theoretically holds the knowledge of 1 trillion parameters, but inference uses only 37 billion. This is the core mechanism enabling GPT-4-level performance at dramatically lower inference cost.

How MoE works

The router determines, for each input token, "which expert is best suited to process this token?"

For example:

"Write Python code" → activates the coding expert cluster
"Explain a historical event" → activates the history/social sciences expert cluster
"Solve a math problem" → activates the math/logic expert cluster

This specialization allows the model to operate efficiently across a wider range of domains.

DeepSeek V4's MoE innovations

DeepSeek V4 added two innovations to standard MoE:

DeepSeekMoE: increases the number of experts while making each one smaller, enabling finer-grained specialization
MLA (Multi-head Latent Attention): compresses the attention cache to reduce memory usage

These two innovations are the core of reducing training costs to roughly one-tenth of traditional large models while maintaining performance.

MoE's limits

Load balancing problem: if load concentrates on certain experts, efficiency drops
Deployment complexity: storing 1 trillion parameters still requires large amounts of VRAM (~600GB at 4-bit quantization)
Training instability: router training is unstable, making large-scale training complex

Path 3: What is synthetic data and why does it matter?

Why is synthetic data needed?

The internet text data used to train frontier LLMs is increasingly exhausted. As of 2026, most high-quality internet text has already been used to train large models.

The solution is synthetic data: existing LLMs generate new training data.

Three types of synthetic data

1. Reasoning process synthesis (Chain-of-Thought synthesis)

Have GPT-5 solve math problems, generating step-by-step solution processes
→ Use these solution processes as training data for smaller models
→ Smaller models learn step-by-step reasoning ability

2. Conversation data synthesis

Use a large model to generate high-quality conversation data across diverse scenarios
→ More varied and consistent conversation patterns than simple chat data
→ Smaller model learns rich conversational ability

3. Domain-specific synthesis

When specialized data for specific domains (medical, legal, coding) is scarce,
generate synthetic data using large model + domain expert review

Synthetic data success stories

Microsoft's Phi-3 series is a flagship synthetic data success. By mass-generating "textbook-quality text" using GPT-4, a 3.8B parameter model outperformed Llama 2-70B.

Qwen3 also uses synthetic data aggressively — particularly step-by-step solution processes generated by GPT-4 in math and coding domains contributed significantly to performance gains.

Synthetic data's limits

Model collapse: training exclusively on synthetic data reduces diversity and accumulates errors
Bias amplification: the teacher model's biases can be passed to the student model intact or amplified
Factual error propagation: incorrect information generated by the teacher model can enter the student model's training data

How the three paths combine: DeepSeek V4's recipe

DeepSeek V4 combined all three technologies:

MoE: 1 trillion total parameters, 37B activated during inference → 90% reduction in inference cost
Synthetic data: math and coding domain synthetic data to strengthen reasoning capability
Indirect distillation: training targets set to match GPT-4-level performance within open-source license scope

Result: API cost 18x cheaper than GPT-4o, performance difference 1–2%.

What does open-source LLMs' catch-up mean for the AI industry?

Why is model commoditization accelerating?

The closer open-source LLMs get to frontier performance, the less differentiation value LLMs themselves carry. This is the "model commoditization" phenomenon we discussed.

OpenAI and Anthropic are shifting their competitive axis from model performance to ecosystem, API convenience, and enterprise trust.

What advantages does self-deploying open-source LLMs give enterprises?

When enterprises deploy open-source models like DeepSeek V4 or Qwen3 on their own servers:

Data doesn't leave for external APIs (security)
No API costs (economics)
Can fine-tune on proprietary data (customization)

These three factors are accelerating enterprises' move to build their own AI infrastructure.

The geopolitical context

DeepSeek V4 and Qwen3 are open-source models made by Chinese companies. The US government has expressed national security concerns about these models. Some US government agencies and defense contractors restrict using Chinese open-source models.

Enterprises should make model selection decisions with awareness of this geopolitical context.

Key action summary

Technology	Principle	Representative examples	Limits
Knowledge distillation	Transfer knowledge from large model to small	Phi series, Qwen3	Cannot exceed teacher
MoE	Activate only a fraction of total parameters	DeepSeek V4, Mixtral	Deployment complexity, VRAM
Synthetic data	AI generates AI training data	Phi-3, Qwen3 coding	Model collapse, bias amplification

FAQ

Q. If open-source LLMs are at frontier level, is there still a reason to use GPT-4 or Claude?

Differences remain: ① performance gap vs. the latest models (GPT-5, Claude Sonnet 4.6) ② safety and alignment research maturity ③ API stability and SLA ④ enterprise support. If cost is the priority, open-source is advantageous. If best performance or enterprise reliability matters, commercial models still hold an edge.

Q. What hardware specs are needed to run DeepSeek V4 locally?

The full model (1 trillion parameters, ~600GB at 4-bit quantization) requires multiple A100/H100 GPUs. For personal experiments, smaller DeepSeek V4 variants (7B, 14B) are the practical option. These can be run locally using tools like Ollama.

Q. Is MoE model inference faster than dense models?

Faster at equivalent performance levels. Compared token-for-token to the same parameter count it may be slower, but because the number of active parameters needed to achieve equivalent performance is far lower for MoE, practical inference speed is faster.

Q. Can models trained entirely on synthetic data be used safely?

Current best practice is to mix synthetic data with real data. Training on 100% synthetic data risks model collapse. Both Phi-3 and Qwen3 mix real internet data with synthetic data.

Q. How do I check open-source LLM licenses?

Llama 3 uses the Meta Llama 3 Community License (commercial use permitted, with conditions); Qwen3 uses Apache 2.0 (commercial use permitted); DeepSeek V4 uses the DeepSeek License (only non-commercial use is free). Always verify the license before commercial use.

Q. How do open-source LLMs perform on non-English language tasks?

Qwen3 shows strength in Asian languages (Chinese, Korean, Japanese). DeepSeek V4's language support has also improved. However, nuance and formal register handling in Korean and other languages still lags behind ChatGPT and Claude.

Q. Can open-source fully catch up to commercial models in the long run?

In the short term (1–2 years), the pattern of trailing 1–2 generations behind will likely continue. Commercial model companies maintain their lead through continued large-scale investment. However, open-source cost-competitiveness will continue increasing.

Q. How should I assess the security concerns around Chinese open-source models?

The US government and some Western companies have raised concerns about backdoors and data collection. Even with published code, complete transparency about training data and training processes isn't available. In enterprise environments where security is critical, US/European open-source models (Llama, Mistral) are recommended.

Update notes

First published: 2026-03-28
Data basis: DeepSeek V4 technical report (January 2026), Qwen3 technical documentation (March 2026), Artificial Analysis benchmarks (March 2026)
Next update: When major open-source model performance updates or new MoE architecture research is published

LLM Architecture Posts

3 Paths Open-Source LLMs Use to Chase the Frontier: Distillation, MoE & Synthetic Data