DeepSeek Architecture Explained: MoE, Multi-Head Latent Attention, and Why It Beats Claude and GPT at 90% Lower Cost

DeepSeek is the most architecturally interesting AI development of 2025 and 2026. It is not just that a Chinese lab built a competitive frontier model — it is that they did it with a radically more efficient architecture that restarted the AI industry's most important debate: whether brute-force scaling was the only path to frontier capability. DeepSeek's answer has been decisively no.

This guide explains DeepSeek's architecture in technical depth — what Mixture-of-Experts means, how the two-type expert system works, what Multi-Head Latent Attention does, how it compares to the dense transformers in Claude and GPT, and what DeepSeek V4's Engram memory introduces. Written for engineering students and technically curious readers.

Dense vs Sparse: The Fundamental Divide

Claude (all versions) and GPT models use dense transformer architectures. In a dense model, every parameter activates for every token. If the model has 100 billion parameters, all 100 billion participate in processing every single word. Inference cost scales linearly with model size — which is why GPT-4 class dense models historically cost $30–60 per million tokens to run.

DeepSeek uses a Mixture-of-Experts (MoE) architecture. The model is divided into specialist sub-networks called experts. For any given input, a learned routing mechanism selects a small subset of these experts — typically 2–4 out of hundreds. The other experts receive no computation at all for that token.

DeepSeek V3 has 671 billion total parameters but activates only approximately 37 billion per token. This delivers the expressive capacity of a 671B model at the inference cost of a 37B model — roughly an 18x reduction per-token compute cost. This is the core reason DeepSeek V3 API access costs $0.27 per million tokens while equivalent dense models cost $15–60.

The Two-Type Expert System

Standard MoE architectures have a routing problem. Unconstrained routing leads to routing collapse — most tokens route to the same popular experts, wasting the rest. Forcing balanced routing prevents routing collapse but prevents the optimal structure where common-knowledge experts are accessed frequently and specialised experts rarely.

DeepSeek's solution, documented in the Epoch AI architectural analysis, is to separate experts into two types. Shared experts always activate for every token — they store universal knowledge that every response needs regardless of topic. Routed experts activate only for specific tokens by the routing mechanism — they store specialised knowledge for specific domains like programming, mathematics, or Chinese language.

This solves routing collapse without artificial balance constraints. The bias-term load balancing — adjusting expert routing bias terms during training rather than using auxiliary loss — maintains approximately even utilisation of routed experts without the performance penalty of traditional methods.

Multi-Head Latent Attention: Memory Compression

Standard transformer attention requires a KV cache that grows linearly with sequence length. At 1 million tokens, this requires enormous GPU memory — making long-context inference extremely expensive with standard attention.

DeepSeek introduces Multi-Head Latent Attention (MLA), which compresses the KV cache into low-rank latent vectors rather than storing full attention states. Instead of full-dimensional key and value vectors per attention head per token, MLA stores one compressed latent vector per token and decompresses it for each head at inference time.

As the Epoch AI analysis notes: this innovation looks obvious in retrospect but requires deep understanding of what attention heads actually do. Standard grouped-query attention achieves memory efficiency by sharing keys and values between heads — but this forces heads to use identical representations, coupling their information. MLA's low-rank compression achieves memory efficiency without this coupling, allowing heads to process the same compressed information in different ways. DeepSeek reports finding beneficial regularising effects from the compression.

How Claude and GPT Architectures Differ

Claude (Anthropic)

Anthropic has not disclosed Opus 4.6 or Sonnet 4.6 architecture in detail. What is documented is that Claude uses a dense transformer. Claude's efficiency advantage over previous generations comes from Adaptive Thinking — dynamic reasoning depth scaling — and from Constitutional AI training methodology that emphasises alignment at a training level rather than through post-hoc filtering. Dense architectures tend to have more uniform quality across all task types because every parameter contributes to every response.

GPT-5.4 (OpenAI)

OpenAI has not disclosed GPT-5.4's full architecture publicly. The configurable reasoning effort API parameter in GPT-5.4 — letting developers dial reasoning depth per request — suggests explicit compute-scaling mechanisms. The computer use capability is implemented as a native model feature rather than a prompt-engineering workaround. GPT-5.4's 33% hallucination reduction likely comes from improved training data quality and reinforcement learning from human feedback on factual accuracy.

Comparing Architectures on What Matters

DeepSeek V4: The Next Generation

DeepSeek V4 (expected Q2 2026) scales to approximately 1 trillion total parameters while maintaining approximately 37B active parameters per token — keeping inference cost comparable to V3 despite the larger total capacity. Two major additions: (1) Engram conditional memory, which separates static fact retrieval (O(1) hash-based DRAM access) from dynamic reasoning, improving Needle-in-a-Haystack accuracy from 84.2% to 97% at 1M tokens. (2) Native multimodality — text, image, and video in one model. Apache 2.0 licence, full commercial rights.