22,580: From GPT-2 to Kimi K3
Seven Years of LLM Architecture Evolution, Explained

Twenty-two thousand five hundred eighty.

That's the number of times GPT-2 (2019, ~124M parameters) fits inside Kimi K3 (2026, 2.8 trillion parameters). In seven years, we made models 22,580× bigger. But is the story really just "bigger"?

In this work-log-style long read, we trace step by step what actually happened over those years — you'll find that a great deal changed, and a surprising amount did not. The original author followed the linearization thread "linear attention → DeltaNet → Gated DeltaNet → Kimi Delta Attention" all the way to Kimi K3; this edition keeps that main thread intact (with all code and derivations) and adds a beginner-friendly math primer, worked-by-hand examples, parameter ledgers, and four side threads: the attention family (MQA/GQA/MLA), state space models (Mamba), MoE sparsity, and training/inference optimization. By the end, someone who knows almost nothing about LLMs can assemble the full knowledge map — the kind where you can actually do the math.

01Prerequisites: Five Math Building Blocks

LLM papers look formula-heavy, but only five building blocks get used over and over: vectors, dot products, matrix multiplication, Softmax, and logarithms. We'll compute each one by hand — something you've computed by hand is something you've truly learned.

1.1 Building blocks one through five, each computed by hand

Block 1: A vector isn't "one meaning" — it's a set of coordinates

Why does one token become thousands of numbers? Because a single scalar can't simultaneously express "part of speech, semantics, tone, entity, position" and dozens of other attributes. The model gives each token a \(d\)-dimensional vector; individual dimensions need not have human-nameable meanings — information is usually distributed across combinations of many dimensions.

Block 2: The dot product answers "how well do two directions match"

Two same-dimensional vectors, multiplied element-wise then summed, give a scalar: \(q \cdot k = \sum_j q_j k_j\).

Block 3: Matrix multiplication is "dot products in bulk"

If \(A\) is \(m \times n\) and \(B\) is \(n \times p\) (the inner \(n\) must match), the product \(AB\) is \(m \times p\). Entry \((i,j)\) of the result is the dot product of row \(i\) of A with column \(j\) of B.

Block 4: Softmax turns arbitrary scores into probabilities

Block 5: Logarithms turn "small probability" into "large penalty"

If the correct answer has probability \(p\), the cross-entropy loss is \(-\log p\). The more confident and correct the prediction, the closer the loss is to 0; the smaller the probability on the right answer, the larger the loss:

1.2 What "predicting the next token" actually optimizes

The training objective of GPT-style models is disarmingly plain: given the preceding tokens, raise the probability of the true next token. Complex abilities don't come from a separate "understand the world" loss — they emerge gradually from compressing the regularities of massive sequences.

First, shift a sentence by one position

During training, all four positions can compute their losses simultaneously (this is why GPU-parallel training works); but each position may only look at itself and to its left — never peek at the target on its right. This constraint is enforced by the "causal mask" of Chapter 02.

The first line factorizes the probability of the whole text into per-step conditionals via the chain rule; the second takes the negative log, turning "maximize a product" into "minimize a sum," which is numerically friendlier.

From token id to probabilities: the complete pipeline

A master map: every layer does only two kinds of mixing — attention-like modules move information between tokens (across positions), and MLP-like modules transform information within each token (within a position). Every architecture innovation in the rest of this article is a replacement for what sits inside these two boxes.

# batch dim omitted; T = sequence length, d = hidden dim, V = vocab size
X = Embedding[token_ids]              # [T] → [T, d]
for layer in layers:
    X = X + Attention(RMSNorm(X))     # mixing across tokens
    X = X + MLP(RMSNorm(X))           # transform within each token
logits = RMSNorm(X) @ W_vocab.T       # [T,d]×[d,V] → [T,V]
prob = softmax(logits, axis=-1)       # each row becomes a vocab distribution
loss = -mean(log(prob[true_next_token]))

Residual connections and normalization: stabilizers for deep networks

If a new branch \(F\) hasn't learned anything useful yet, the model can still pass information and gradients along the "+x" express lane; deep networks are therefore much easier to train. This is the famous Pre-Norm residual structure.

The MLP: two (or three) linear transforms

The original GPT-2 used a two-layer MLP: expand from \(d\) to \(d_{ff}\), apply GELU, project back to \(d\). Modern models mostly use the three-way SwiGLU structure:

\(\odot\) is element-wise multiplication. The gate branch decides which expanded dimensions pass through, the up branch supplies content, and down projects back to model width. Ignoring biases, the three weight matrices hold \(d \times d_{ff}\), \(d \times d_{ff}\), and \(d_{ff} \times d\) parameters — about \(3 d d_{ff}\) in total. This single formula will later explain exactly where Kimi K3's trillion parameters come from (full reconciliation in §11.2).

02Ground Zero: Dissecting GPT-2

To understand every improvement that follows, we must first strip the baseline model to the bone. GPT-2 is a canonical decoder-only architecture — nearly every mainstream LLM today (GPT, Llama, Qwen, DeepSeek, Kimi) belongs to this family. Its entire forward pass fits in under 15 lines of code:

def forward(self, idx, pos):        # idx: (b, t) token ids; pos: (t,)
    tok_emb = self.transformer.wte(idx)  # (b, t) → (b, t, n_embd)
    pos_emb = self.transformer.wpe(pos)  # (t,) → (t, n_embd)
    x = self.transformer.drop(tok_emb + pos_emb)  # (b, t, n_embd)
    for block in self.transformer.h:
        x = block(x)                     # 12 Transformer blocks, shape unchanged (b, t, n_embd)
    x = self.transformer.ln_f(x)         # final LayerNorm, (b, t, n_embd)
    logits = self.lm_head(x)             # (b,t,n_embd)×(n_embd,V) → (b, t, vocab_size)
    return logits

2.1 From embeddings to output: one complete forward pass

The input text is first split by a tokenizer into a token sequence (GPT-2 uses BPE, vocab ≈ 50,257). Each token looks up a 768-dim vector (token embedding), and the vector of its position is added (position embedding) — note that GPT-2's positional encoding is a learned absolute position embedding, a trainable lookup table just like the token embedding. This would later be completely overhauled (see §3.2).

After the embeddings are added, the data flows through 12 structurally identical Transformer blocks. Zooming into one block:

class Block(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.ln_1 = LayerNorm(config.n_embd, bias=config.bias)
        self.attn = CausalSelfAttention(config)
        self.ln_2 = LayerNorm(config.n_embd, bias=config.bias)
        self.mlp = MLP(config)

    def forward(self, x):               # x: (b, t, n_embd)
        x = x + self.attn(self.ln_1(x))   # attention sublayer + residual → (b, t, n_embd)
        x = x + self.mlp(self.ln_2(x))    # MLP sublayer (up to 4×n_embd, then back) → (b, t, n_embd)
        return x                      # (b, t, n_embd)

2.2 Causal attention, computed by hand

The heart of each block is Causal Self-Attention. Its code is startlingly short too:

B, T, C = x.size()  # batch, sequence length, embedding dim; x: (B, T, C)

# one linear projection computes q, k, v at once, then split into heads
q, k, v = self.c_attn(x).split(self.n_embd, dim=2)  # (B,T,C)→(B,T,3C), split into three (B, T, C)
k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs)
q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs)
v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs), hs = C/nh is head dim

# attention scores: each query dotted with all keys, divided by √d_head
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))  # (B,nh,T,hs)×(B,nh,hs,T) → (B, nh, T, T)
att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))  # causal mask, (B, nh, T, T)
att = F.softmax(att, dim=-1)  # (B, nh, T, T), each row sums to 1
y = att @ v          # (B,nh,T,T)×(B,nh,T,hs) → (B, nh, T, hs)
y = y.transpose(1, 2).contiguous().view(B, T, C)  # reassemble heads → (B, T, C)
y = self.resid_dropout(self.c_proj(y))  # output projection → (B, T, C)
return y

Condensed into a formula:

Rather than memorizing the formula, let's actually compute one pass. Suppose the projections have produced Q/K/V for three tokens (2-dim, single head). We only want the output of token 3, i.e. we dot \(q_3 = [0,1]\) with each of the three keys:

The causal mask: no peeking at the future, even in training

For row \(t\), only positions \(1 \dots t\) are visible. Scores at future positions get \(-\infty\) added, which Softmax turns into weight 0:

This lets training compute all positions in parallel while preserving the same information constraint as token-by-token generation. What runs in parallel is the matrix math — early positions never see the future. This is the mathematical guarantee of "autoregressive generation," and the dividing line between decoder-only models and BERT-style encoders.

Multiple heads are not computing the same thing repeatedly

Each head has its own projection matrices, so it can learn a different matching subspace (syntax, coreference, positional proximity, etc.). With total hidden dim \(d = h \cdot d_h\), concatenation returns to d dimensions, and \(W_O\) then mixes the heads. The number and organization of heads becomes the protagonist of Chapter 03.

Where the quadratic complexity comes from

\(QK^\top\) produces a \(T \times T\) score matrix: each of T queries compares with each of T keys, each dot product costs ≈ d, so the rough compute is \(O(T^2 d)\), and storing the attention weights explicitly is \(O(T^2)\). Feel the horror of "quadratic" in this table:

2.3 KV Cache: the first engineering compromise

After the last Transformer block outputs the hidden-state matrix, the language model head (LM head) maps it to vocab-sized logits. In autoregressive decoding, each step needs only the logits at the last position to sample the next token.

This exposes a natural inefficiency of decoder-only generation: the model computes full representations for every input position, yet each decoding step consumes only the last position's logits. Without any caching, generating the next token would redo all of that computation verbatim.

KV Cache comes from a plain observation: after appending the newly generated token to the input, the key and value projections of all previous tokens are bit-for-bit identical — the causal mask guarantees that past representations are unaffected by the future. So store their K and V vectors; the next step only needs to compute QKV for the new token and attend against every cached K and V. Compute per step drops from \(O(N^2)\) to \(O(N)\).

That storage is the KV Cache. It keeps the K/V vectors of the previous \(N-1\) tokens, at \(2 \times N \times n_{layer} \times n_{head} \times d_{head} \times 2\) bytes (FP16), growing linearly with sequence length — quickly large enough to turn inference into a memory-bandwidth bottleneck rather than a compute bottleneck. Remember this sentence: from GQA to MLA to linear attention, almost the entire evolutionary line that follows is a war against "KV Cache too big."

From MQA/GQA (store fewer copies) to MLA (compress each copy) to linear attention (drop the list entirely) — seven years of architecture innovation has been a search for balance points between the two ends of this table.

2.4 The parameter ledger: where 124M comes from

"Parameters" are the numbers updated during training and read during inference. Once you see the shape of every weight matrix, parameter counts stop being a mysterious vendor total and become an addition problem you can audit. A dense Transformer has four ledgers:

GPT-2's configuration: ~50K vocab, 12 blocks, 12 heads, 768-dim embeddings:

vocab_size: int = 50304  # GPT-2 vocab is 50257, padded up to a multiple of 64 for compute
n_layer: int = 12
n_head: int = 12
n_embd: int = 768

Total: ~124 million parameters. Kimi K3 has 2.8 trillion — one K3 holds about 22,580 GPT-2s.

03Interlude I: The Attention Family Tree

The original's main thread jumps straight into "linear attention," but before that, industry actually took another path first: don't change attention's math — change its storage and engineering. This side thread gave us GQA, RoPE, RMSNorm, and FlashAttention, standard equipment in every mainstream model today. We must fill it in.

3.1 MHA → MQA → GQA → MLA: the siege on KV Cache

GPT-2 uses Multi-Head Attention (MHA): 12 heads, each with its own independent Q, K, V. The problem is the KV Cache — every head must store a K and a V for every historical token. 12 layers × 12 heads × 2 (K and V) × 64 dims × per token — memory explodes on long sequences.

The evolutionary chain is textbook-clear:

• MQA (Multi-Query Attention, Shazeer 2019): all query heads share a single K and V. The KV Cache is divided by the head count (12× compression), decoding throughput jumps — but with a perceptible quality loss.
• GQA (Grouped-Query Attention, 2023, popularized by Llama 2): the compromise — split Q heads into groups, each group sharing one K/V. Llama 2 70B uses \(h=64\) Q heads with \(g=8\) KV groups, compressing the cache 8× with almost no quality loss. GQA then became the default for 2023–2024 open models (Llama, Qwen, Mistral).
• MLA (Multi-head Latent Attention, DeepSeek-V2, 2024): a different idea — instead of sharing K/V, jointly low-rank compress all heads' K/V into a small latent vector, store that, and decompress on use. Higher compression than GQA, and paradoxically better quality. This is core technology of DeepSeek-V2/V3 and the Kimi family; §10.2 dissects it in detail.

3.2 Position encoding: the victory of RoPE

GPT-2 uses learned absolute position embeddings, with two hard flaws: the longest sequence seen in training is the capability ceiling, and position information is entangled with content information. The evolutionary path:

• Sinusoidal encoding (original 2017 Transformer): generated by a fixed formula, no learning needed, slightly better extrapolation — still fragile.
• RoPE (Rotary Position Embedding, Su Jianlin 2021): encode position as a rotation of 2-D subspaces of the Q/K vectors — the query at position \(m\) is rotated by \(m\theta\). All of Llama, Qwen, DeepSeek, and Kimi use RoPE, one of the few "zero-controversy" technical consensuses of recent years.
• Length-extrapolation patches: RoPE fails when extrapolated directly beyond trained lengths, so a family of "length-extension" techniques appeared — Position Interpolation, NTK-aware scaling, YaRN — letting a 4K-trained model serve 128K or longer.
• ALiBi (2022): instead of adding position to embeddings, add a distance-proportional penalty directly to attention scores; excellent extrapolation (adopted by BLOOM).
• NoPE / emergent causal position: another interesting finding — decoder-only models learn some position sense even with no position encoding at all; and the linear-attention/RNN architectures of Chapter 04 have position built into the recurrence by construction.

3.3 Normalization and activations: the quiet foundation

These changes aren't sexy, but each one saved real money:

3.4 FlashAttention: same math, new world

Tri Dao's FlashAttention (2022) is perhaps the single most industry-impactful paper of the era. It changed nothing about attention's mathematical definition — its output is bit-equivalent to the standard implementation. What it changed is how the computation is organized on the GPU.

The key insight: the bottleneck of standard implementations is not FLOPs but memory IO. A naive implementation materializes the full \(N \times N\) attention-score matrix to HBM (high-bandwidth memory), reads it back for softmax, writes it back, reads it again to multiply by V — at 8K sequence length that intermediate matrix is already several GB. GPU SRAM (on-chip cache) is an order of magnitude faster but tiny (~200KB per SM). FlashAttention's approach:

1. Tiling: cut Q/K/V into blocks that fit in SRAM;
2. Online softmax: while sweeping K/V blocks, maintain a running maximum and normalization factor, completing exact softmax in one pass — the \(N \times N\) matrix is never materialized;
3. Recomputation: in backprop, don't store the attention matrix — recompute it on demand. More FLOPs, less IO, faster overall.

FlashAttention-2 (better parallelism partitioning) and FlashAttention-3 (async pipelines for Hopper GPUs) kept squeezing the hardware. Note one chain reaction: by making full-length attention engineering-cheap, FlashAttention temporarily squeezed linear attention out of the market — until the KV-Cache bandwidth bottleneck in long-sequence, high-concurrency decoding brought the linearization route back to the table. That is exactly the historical backdrop of Chapter 04.

04Linear Attention: Bring Back the RNN

Now back to the original's main thread. The first route that tried to flip the \(O(N^2)\) table mathematically was linear attention, in 2020.

Softmax attention applies its nonlinearity (the exponential) after the \(q \cdot k\) dot product, which tightly couples every query with every key — you must compute the full \(N \times N\) score matrix before normalizing. Linear attention swaps the order: first apply a feature map (e.g. ELU+1) to q and k separately, then let them multiply:

The magic of this swap is associativity: \(\phi(q)\cdot(\phi(k)^\top v)\) can evaluate \(\phi(k)^\top v\) first — and the \(\phi(k_j)^\top v_j\) of all historical tokens can be accumulated into one fixed-size \(D \times D\) matrix \(S\). The whole thing becomes an RNN: each new token adds a little something into the state \(S\); reading out is a single \(q \cdot S\) matrix-vector product. The causal version can be written as a recurrence:

def forward(self, x, mask=None, cache=None):
    # x: (b, t, d)
    b, t, d = x.shape
    d_head = d // self.num_heads          # head dim dh = d/h
    qkv = self.qkv_proj(x)                # (b,t,d) → (b, t, 3d)
    q = qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2)      # (b, h, t, dh)
    k = qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2)   # (b, h, t, dh)
    v = qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2)    # (b, h, t, dh)

    k = F.elu(k) + 1          # feature map: keeps things non-negative, (b, h, t, dh)
    k = k.transpose(-1,-2)    # (b, h, dh, t)
    q = F.elu(q) + 1          # (b, h, t, dh)

    S, z = cache if cache is not None else (0.0, 0.0)  # S: (b,h,dh,dh); z: (b,h,dh,1)
    S = S + k @ v             # (b,h,dh,t)×(b,h,t,dh) → (b,h,dh,dh), fixed D×D regardless of length
    z = z + k                 # (b,h,dh,1), normalizer accumulator

    o = q @ S                 # (b,h,t,dh)×(b,h,dh,dh) → (b,h,t,dh), one matvec readout
    denom = q @ z             # (b,h,t,1)
    o_scaled = o / denom      # (b,h,t,dh)
    ...
    cache = (S, z)            # constant shape, never grows with t
    return o_proj, cache      # o_proj: (b, t, d)

What's the price? Expressiveness. Clearest when you decompose attention into three steps:

1. Make the qk scores non-negative — softmax uses the exponential; linear attention uses ELU+1;
2. Normalize by the total score;
3. Take a weighted average of the values.

All three steps survive, but the "non-negativizing" function in step one went from exponential to ELU+1 — a lower-expressiveness approximation of the softmax kernel. The exponential can produce sharp, winner-take-all attention distributions (precise retrieval of a single token); ELU+1 produces flatter, blurrier distributions. The actual accuracy loss depends on architecture and task, but on tasks that require "fishing one precise detail out of a long context," naive linear attention's disadvantage is real.

05DeltaNet: A Whiteboard That Corrects Itself

A finite-capacity cache inevitably faces overwriting and interference. In linear attention, token \(i-1\)'s state has no slot of its own — it got added into the same \(D \times D\) matrix. When a new query wants to read back some early token's independent representation with precision, it no longer can.

This "addition" is both the source of efficiency and the source of disaster: updating the cache by accumulation rather than concatenation is what keeps it from O(N) bloat; but accumulation is also what lets information pollute each other. Schlag et al. put it perfectly in Linear Transformers Are Secretly Fast Weight Programmers:

"When sequence length exceeds storage capacity, the model enters an 'over-capacity' regime. To function there, the model should learn to interact dynamically with memory contents, selectively deciding which key-value associations to keep and which to delete. A purely additive rule is unsuited for this purpose… endlessly adding new associations to a finite memory inevitably hits the limit."

5.1 First, understand: an associative-memory whiteboard

Look at how a single association is stored and retrieved. Write \(S = k^\top v\) (outer product); read with the same key: \(k(k^\top v) = (kk^\top)v = \|k\|^2 v\). That is, the readout is exactly the value times the squared norm of the key — normalize k to unit length (or divide by the norm after reading) and you get v back losslessly.

Q here is a learnable pointer: \(W_q\) and \(W_k\) read the same residual stream, and the query for a fact will point toward the key direction under which that fact was originally written. This machinery descends directly from last century's Hopfield networks — Transformer attention was always a modern associative memory.

5.2 The delta rule: erase first, then write

The delta rule (the modern incarnation of the Widrow-Hoff rule) updates like this:

1. First read the cache with the current key: \(v_{old} = k_i S\), see what's stored at this location now;
2. Compute the delta: the difference between the new value you want and the old value, scaled by write strength \(\beta\): \(u_i = \beta_i (v_i - v_{old})\);
3. Write the delta back: \(S \leftarrow S + k_i^\top u_i\).

def forward(self, x, mask=None, cache=None):
    # x: (b, t, d)
    b, t, d = x.shape
    ...
    q = F.normalize(F.silu(q), dim=-1)   # (b,h,t,dh), unit length
    k = F.normalize(F.silu(k), dim=-1)   # (b,h,t,dh), normalized: lossless readout
    beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1)  # (b,1,t,1), new: per-token write strength

    S = cache if cache is not None else 0.0   # (b, h, dh, dh)

    v_old = k @ S                        # (b,h,t,dh)×(b,h,dh,dh) → (b,h,t,dh), read old info at this key
    u = beta * (v - v_old)               # (b,h,t,dh), delta: write only what's truly new
    S = S + k.transpose(-1, -2) @ u      # (b,h,dh,t)×(b,h,t,dh) → (b,h,dh,dh), same outer-product write

    o = q @ S                            # (b,h,t,dh)×(b,h,dh,dh) → (b,h,t,dh), readout (no normalizer needed)
    o = o.transpose(1, 2).contiguous().view(b, t, d)  # (b, t, d)
    return self.o_proj(o), S             # (b,t,d); cache S: (b,h,dh,dh)

Old information is removed in a targeted way and new information is written in its place — the cache turns from an "append-only ledger" into an "editable whiteboard." \(\beta\) is input-dependent (a sigmoid gate): the model learns for itself how hard each token should rewrite memory.

06Parallelizing DeltaNet: The Chunking Trick

6.1 The problem: sequential dependency at prefill

Token-by-token recurrence is natural at inference, but during training/prefill, if you run T tokens through the delta rule one at a time, the GPU just sits there watching:

S = torch.zeros(b, h, dh, dh) if cache is None else cache  # (b,h,dh,dh)
outs = []
for i in range(t):                # purely sequential loop, T small matrix ops
    k_i = k[:, :, i:i+1]          # (b,h,1,dh)
    v_i = v[:, :, i:i+1]          # (b,h,1,dh)
    b_i = beta[:, :, i:i+1]       # (b,1,1,1), scalar write strength
    v_old = k_i @ S               # (b,h,1,dh)×(b,h,dh,dh) → (b,h,1,dh)
    u_i = b_i * (v_i - v_old)     # (b,h,1,dh)
    S = S + k_i.transpose(-1, -2) @ u_i   # (b,h,dh,1)×(b,h,1,dh) → (b,h,dh,dh), write
    outs.append(q[:, :, i:i+1] @ S)       # (b,h,1,dh)
o = torch.cat(outs, dim=2)        # (b,h,t,dh)

Unlike standard attention: the delta rule must "read–correct" at every key, and step \(i\) depends on the state corrected at step \(i-1\). The path to parallel matrix multiplication is anything but obvious. In fact, even without the delta correction, naive linear-attention prefill is equally sequential (\(S\) accumulates step by step).

6.2 Chunking: stitching two orders of association

The way out is chunking. Cut the sequence into chunks of length C (in practice C is often 64 or 128, because tensor-core instructions are most efficient at that granularity, e.g. NVIDIA's UMMA):

• Inside a chunk: compute in the order \(q(k^\top v)\) — scores first, then masked weighting, i.e. "true attention" mode; intra-chunk complexity \(O(C^2)\);
• Across chunks: go in the order \((k^\top v)q\) — at each chunk's end, fold the whole chunk's contribution into the state \(S\); the next chunk reads from the state with one matrix multiply, i.e. "recurrent" mode.

S = torch.zeros(b, h, dh, dh) if cache is None else cache  # (b,h,dh,dh)
outs = []
for i in range(t // C):                       # only t/C iterations, each a big matmul
    q_c = q[:, :, i*C:(i+1)*C]          # (b,h,C,dh)
    k_c = k[:, :, i*C:(i+1)*C]          # (b,h,C,dh)
    v_c = v[:, :, i*C:(i+1)*C]          # (b,h,C,dh)

    o_prev = q_c @ S                            # (b,h,C,dh)×(b,h,dh,dh) → (b,h,C,dh), all history before this chunk
    attn = (q_c @ k_c.transpose(-1,-2)).tril()  # (b,h,C,C), masked intra-chunk attention
    o_curr = attn @ v_c                         # (b,h,C,C)×(b,h,C,dh) → (b,h,C,dh)
    o = o_prev + o_curr                         # (b,h,C,dh)

    S_new = k_c.transpose(-1,-2) @ v_c          # (b,h,dh,C)×(b,h,C,dh) → (b,h,dh,dh), fold the chunk into the state
    S = S + S_new                               # (b,h,dh,dh)
    outs.append(o)
o = torch.cat(outs, dim=2)              # (b,h,t,dh)

The cost decomposition is elegant: a fixed part \(2Ld^2\) (state maintenance, independent of C) + a growing part \(2LCd\) (the score matrices on the diagonal). C = L degenerates to standard attention (the second term becomes \(2L^2d\), quadratic); C = 1 is pure linear attention. Intermediate C trades a little intra-chunk compute for hardware utilization — C=1 is cheapest in pure FLOPs, but not in wall-clock time: GPUs only run fast when work maps to large matrix multiplies.

6.3 Squeezing the delta correction into chunks: WY reparameterization

Applying the same chunking idea directly to the delta rule hits a wall: each key's correction depends on the state at that moment, while chunk-parallelism requires all corrections within a chunk to be computed at once:

v_old = k_i @ S               # (…,1,dh)×(…,dh,dh) → (…,1,dh)
u_i = b_i * (v_i - v_old)     # (…,1,dh), every u_i depends on S after all prior corrections

We would have to pass through every intermediate state in order to know what to subtract — without mathematical reparameterization, no parallelism. The authors' approach: expand the recurrence

and discover that \(S_T\) can be written as the initial state left-multiplied by a string of \((I - \beta k k^\top)\) products, plus a weighted sum of \(v k^\top\) terms — exactly the WY representation (the compact form of products of Householder transforms in numerical linear algebra). The corrections \(u\) can be solved all at once by forward substitution on a C×C lower-triangular matrix. The full four-step derivation from recurrence to expansion is in Appendix B; here we land directly on the code — the chunked code computes all C deltas of a chunk at once:

def chunk_delta_rule_forward(Q, K, V, beta, C):
    # Q,K,V: (L, d); beta: (L,); C: chunk length
    L, d = Q.shape
    # chunk
    Q, K, V = map(lambda x: x.reshape(-1, C, d), [Q, K, V])  # each (L/C, C, d)
    beta = beta.reshape(-1, C)            # (L/C, C)
    K_beta = K * beta.unsqueeze(-1)       # (L/C, C, d)
    V_beta = V * beta.unsqueeze(-1)       # (L/C, C, d)

    # vectorized forward substitution: invert (I + strictly lower triangular)
    T = -(K_beta @ K.t()).tril(-1)        # (L/C, C, C), strictly lower triangular
    for i in range(1, C):
        T[i, :i] = T[i, :i] + (T[i, :, None] * T[:, :i]).sum(-2)
    T += torch.eye(C)                     # (L/C, C, C), lower triangular, unit diagonal
    W = T @ K_beta        # (L/C,C,C)×(L/C,C,d) → (L/C, C, d), corrected keys
    U = T @ V_beta        # (L/C, C, d), corrected values (all deltas included)

    # chunk-parallel main loop
    S = torch.zeros(d, d)                 # (d, d)
    O = torch.empty_like(V)               # (L/C, C, d)
    for i in range(L // C):
        q_i, k_i, w_i = Q[i], K[i], W[i]  # each (C, d)
        u_i = U[i] - w_i @ S              # (C,d) − (C,d)×(d,d) → (C, d), the chunk's corrections at once
        o_inter = q_i @ S                 # (C, d), history contribution (recurrent mode)
        A_i = (q_i @ k_i.t()).tril()      # (C, C), intra-chunk attention scores
        o_intra = A_i @ u_i               # (C,C)×(C,d) → (C, d), intra-chunk contribution (attention mode)
        S += k_i.t() @ u_i                # (d,C)×(C,d) → (d, d), update the state
        O[i] = o_intra + o_inter          # (C, d)
    return O.reshape(L, d)                # (L, d)

With this we reach the first milestone comparison — the standard multi-head attention Transformer vs the DeltaNet Transformer:

07Interlude II: State Space Models and Mamba

Half of the Gated DeltaNet in the next chapter descends from Mamba. So before the "forget gate," we must meet linear attention's parallel universe — State Space Models (SSM).

7.1 From control theory to deep learning: the math of SSMs

The state space model is an antique of control theory: a continuous-time linear time-invariant system

where \(x(t)\) is the input signal (in an LLM, one dimension of a token embedding), \(h(t) \in \mathbb{R}^N\) is the hidden state, \(A\) the state-transition matrix, \(B\) and \(C\) the input/output projections. To handle discrete token sequences, discretize with Zero-Order Hold:

The discretized recurrence is a linear RNN — per-step cost independent of sequence position, which is how SSMs achieve \(O(L)\) total complexity (instead of \(O(L^2)\)). Its resemblance to linear attention's \(S_t = S_{t-1} + k_t^\top v_t\) is no coincidence: both compress all of history into a fixed-size hidden state — they just speak different dialects.

Key milestones of the SSM family:

• S4 (Gu et al., 2021): the first SSM to beat Transformers on very long sequences. The core: initialize \(A\) with the HiPPO matrix (a mathematically optimal "history compression basis") and train in parallel via diagonalization + convolution tricks. But S4's transition matrix is fixed — it doesn't change with input, like a memory machine that can't adjust to content.
• Mamba (Gu & Dao, 2023): make \(B\), \(C\), and the discretization step \(\Delta\) all functions of the input — "selective." The model can decide per content whether to remember carefully or ignore this step. The cost: the convolutional parallel form is lost, so a hardware-aware parallel scan algorithm was invented to run the recurrence in SRAM. Mamba was the first linear-time architecture to match same-size Transformers on language modeling.
• Mamba-2 / SSD (2024): revealed the strict duality between SSMs and linear attention — a scalar-gated SSM is "linear attention with a forget gate." This insight unified the two parallel universes and paved the road directly to Gated DeltaNet (next chapter).

7.2 Other runners on the same track

08Gated DeltaNet: Learning to Forget

Back to the main thread. DeltaNet gave us the ability to precisely edit the cache: every arriving key carrying a new fact can read the old content at its location and replace it in a targeted way. But it has a blind spot —

It can only forget "associations that have a replacement." Want to clear a batch of old memories when the topic changes? Want unimportant facts to decay over time to make room? The delta rule can't: its eraser wipes only the small patch under one key at a time.

Conversely, if we were doing purely additive linear attention, adding "forgetting" is actually easy — it takes only a single scalar parameter:

S_old = cache                          # (b,h,dh,dh)
S_new = k @ v                          # (b,h,dh,t)×(b,h,t,dh) → (b,h,dh,dh)
# cache = S_old + S_new          # the old pure accumulation
cache = alpha * S_old + S_new    # alpha: (b,1,t,1) scalar gate → (b,h,dh,dh), decay old state, then write at full strength

This is exactly Mamba-2's contribution: decay the old cache first, then write the new content at full strength, preventing unbounded state inflation. But uniform decay has an obvious flaw — it doesn't distinguish how important associations are. To forget one specific association, every association must age by the same proportion. The delta rule is the exact opposite: it can update a single fact surgically, but can't batch-decay the rest.

Gated delta rule = Mamba's gating × the delta rule's precision. Introduce the parameter \(\alpha_t\): at \(\alpha = 1\) it degenerates to the pure delta rule; at \(\alpha = 0\) it wipes memory clean. The update:

The implementation reuses Chapter 06's WY reparameterization; the math barely changes — just one more data-dependent scalar between 0 and 1 controlling the decay of the previous state. The difficulty is again engineering: the chunk-parallel form must handle cumulative decay —

09KDA and Kimi Linear

At this stage, researchers began trying hybrid architectures — interleaving multiple attention forms in one model, e.g. mixing Gated DeltaNet with Mamba, or linear layers with full-attention layers. In 2025, Kimi Linear pushed this path to a landmark conclusion:

9.1 KDA: fine-grained gating

Kimi Delta Attention's (KDA) key improvement over Gated DeltaNet is upgrading the forgetting from a scalar to a per-channel vector: no longer decaying the entire state matrix with a single α, but learning an independent decay rate for every feature channel.

The update rule's skeleton is unchanged, but the code gains one reshape:

Fine-grained gating is not simply "adding parameters." It has a precise mathematical function: installing a tiered recycling system on a capacity-limited memory — an independent decay rate per channel is equivalent to soft memory-partition management in feature space. This is exactly the engineering solution to Chapter 05's "over-capacity aliasing" problem.

9.2 Kimi Linear's complete recipe

Side by side with the DeltaNet Transformer, Kimi Linear introduces three major changes:

1. Hybrid architecture: MLA (Multi-head Latent Attention) layers interleaved in — 3 KDA layers per 1 full-attention layer;
2. MoE replaces the MLP: feed-forward networks swapped for mixture-of-experts layers;
3. Expanded α projection: extra capacity for the DeltaNet mechanism via the alpha projection (the fine-grained gating above).

Details of MLA and MoE are deferred to Chapter 10. Here, stress one important argument of the original: this is not blind ingredient-stacking. Every added capacity serves a concrete mathematical purpose — per-channel α gives fine control over memory decay; periodic MLA restores the precise retrieval that linear states necessarily lose; MoE adds knowledge capacity without adding per-token compute.

Scaling laws still hold, but capacity must be added in the right places, in forms the system can digest. Every architecture on this evolutionary line is targeted capacity expansion for a specific defect of the previous system.

10Interlude III: MoE and MLA in Depth

The two pillars of Kimi Linear and Kimi K3 — MoE and MLA — got only passing mention in the original. But they were the main battleground of the 2024–2026 LLM arms race (DeepSeek's rise rests almost entirely on these two), and each deserves a section.

10.1 MoE: the art of sparsity

Recall Chapter 02's parameter breakdown: in GPT-2, about two-thirds of the parameters live in the MLP. The MLP is considered the model's "knowledge warehouse" (attention routes information; the MLP stores facts). So here's a tempting idea: make many copies of the MLP, but let each token use only a few of them — parameter count (knowledge capacity) explodes while per-token compute barely changes. That is Mixture-of-Experts.

Key design points and history:

• Origins: Shazeer et al.'s Sparsely-Gated MoE in 2017 (the LSTM era!); in 2021 Google's Switch Transformer (simplified to Top-1 routing) and GShard took MoE to ten-thousand-GPU scale.
• Load balancing: routers tend to "despise the poor" — a few experts get picked frantically while the rest starve, and training collapses. The classic fix is an auxiliary load-balancing loss forcing uniformity, but it hurts model quality; DeepSeek-V3 pioneered the auxiliary-loss-free route: give each expert a dynamic bias term that affects only routing decisions and never enters gradients — balance and quality at once.
• Fine-grained + shared experts (DeepSeekMoE, 2024): cut experts smaller and more numerous (greater combinatorial flexibility), and designate a few shared experts that handle common knowledge for all tokens while routed experts learn only specialized knowledge — mitigating knowledge redundancy. Kimi K3's 898 experts (2 shared + 896 routed) is the jumbo version of this idea.
• The open-source detonation: Mixtral 8×7B (2023) proved even a mid-size MoE can crush same-tier dense models; after that, DeepSeek-V3 (671B / 37B activated), Qwen3-MoE, and Kimi K2 (1T / 32B activated) made MoE the default form of frontier models.
• The cost: memory must hold all experts (parameters don't shrink), training communication is complex (expert-parallel AllToAll), and expert utilization is poor at small inference batches. What MoE saves is FLOPs, not memory.

10.2 MLA: distilling the KV Cache

§3.1 said MLA is the newest weapon in the siege on KV Cache; now let's take it apart. GQA's idea is "let Q heads share K/V"; MLA (DeepSeek-V2, 2024) is more radical: jointly low-rank-compress all heads' K and V into a single latent vector \(c_t\), and cache only that, decompressing into each head's K/V on use.

Adding multi-head latent attention's exploitation of "redundancy between heads," MLA achieves an anomalous result: a cache far smaller than GQA's, with quality better than MHA's.

11Kimi K3: The 22,580× Endpoint (For Now)

All the threads converge here. Kimi K3's language backbone is a scaled-up Kimi Linear: 23 four-layer macrocycles — within each macrocycle, 3 layers use Kimi Delta Attention and the 4th uses MLA; the first layer uses a dense feed-forward network, everything else becomes latent-space MoE.

Compared with Kimi Linear, the change list looks unremarkable at first:

• Substantially larger scale (to 2.8T parameters)
• Blockwise AttnRes every 12 layers (detailed in Chapter 12)
• Query LoRA and output gating for MLA
• Latent-space MoE
• The SiTU activation function
• Gated MLA

The division of labor is the same as ever: KDA provides constant-state recurrent memory, while periodic MLA layers preserve full softmax retrieval over the context. One diagram to rule the whole model:

11.1 Three direct changes: Gated MLA, latent-space MoE, SiTU

Gated MLA: decides how much of what MLA retrieves may enter the residual stream — a gate projected from the input, multiplied element-wise with the attention output. In essence it puts a faucet on the "precise retrieval channel": retrieval is accurate, but how much of it to let into the trunk is decided by context.

Latent-space MoE: in a traditional MoE, the router sends each token to several experts by dot-product similarity. Kimi K3 has 898 experts: 2 shared experts process every token, and the router picks 16 of the remaining 896 for each token. K3's further innovation is letting experts work in a compressed latent space:

The SiTU activation: K3 also replaces the activation inside experts. Instead of the old "expand → SiLU → element-wise gate multiply → contract" route, it uses SiTU:

d = x.shape[-1] // 2                              # x: (..., 2d), first half gate, second half up
gate = x[..., :d].to(torch.float32)               # (..., d)
up = x[..., d:].to(torch.float32)                 # (..., d)
situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate)  # (..., d)
if self.linear_beta is not None:
    up = self.linear_beta * torch.tanh(up / self.linear_beta)            # (..., d)
return (situ_a * up).to(x.dtype)                # (..., d)

Note the learnable \(\beta\) inside the tanh term: the saturation region of the gate becomes learnable — the model can tune the "hardness" of the activation per layer.

The remaining changes: query LoRA, output gating, AttnRes

MLA gets LoRA low-rank adaptation on the query side and gating on the output, plus Blockwise Attention Residuals every 12 layers. AttnRes adds only ~2% inference latency, in exchange for two important gains: selective retrieval of early representations (mitigating residual dilution and hidden-state magnitude growth), and a 1.25× compute advantage (stronger at equal compute).

The last point is worth pausing on: AttnRes and MLA attack the same underlying limitation from different directions. KDA layers have constant state and necessarily discard information; MLA backstops with lossless retrieval along the sequence dimension (token context), while AttnRes backstops with selective retrieval along the depth dimension (representations of early layers). One patches horizontal memory, the other vertical memory.

11.2 The ledger: how 1T becomes 2.8T

In Chapter 02 we audited GPT-2's 124M; now we apply the same method to K3. First answer a common confusion: K3 doesn't have several times more layers than K2 — how did parameters grow from 1T to 2.8T? The answer hides in a three-factor MoE product:

12AttnRes: Attention Across Depth

(The original credits @chloey3k for help with this section.) In every forward pass, the input traverses a stack of layers — each layer an attention block (KDA or MLA) plus an MLP/MoE block. Traditionally, each layer's input is the equal-weighted sum of the original embedding and all previous layers' outputs:

where \(h_i\) is layer \(i\)'s input, \(h_1\) is the current token's embedding, and \(f_i(h_i)\) is layer \(i\)'s output. The problem: no selectivity. Different kinds of layers receive the same "hodgepodge" state even if they want to weight history differently; and because the recurrence is purely additive, later layers must learn ever-larger outputs to influence the accumulated residual — causing both "residual dilution" (early information drowned out) and continuously growing hidden-state magnitudes, which destabilizes training.

AttnRes's approach: stop treating everything equally — multiply each term of the sum by a dedicated weight:

Each weight \(\alpha_i\) is computed from a query-key dot product: queries are learned per layer, keys and values come from earlier residual-stream states; scores are normalized to sum to 1, then combined. The model no longer has to condition only on the immediately preceding layer — every layer can use its learned query to retrieve, from the entire depth history, the representations most useful to its current computation. This generalizes "attention" from the sequence dimension (between tokens) to the depth dimension (between layers).

Applied at block granularity: a "block" is the element-wise accumulation of attention and MLP outputs over 12 decoder layers, stored as a single depth representation for later AttnRes mixing. Doing residual attention at every layer is too expensive; doing it only at fixed block boundaries captures most of the benefit at small cost — in K3, boundaries are 12 layers apart, and the 23 four-layer macrocycles yield 8 AttnRes blocks, which also improves inference speed.

The following is probably the most important part of the block_attn_res function:

V = torch.stack(blocks + [partial_block])       # [N+1, B, T, D] all historical blocks
K = norm(V)                                     # [N+1, B, T, D], normalized for scoring
logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K)  # [N+1, B, T], one score per block
h = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V)        # softmax over the block dim → weighted sum (B, T, D)
return h                                        # (B, T, D)

Four lines: stack all historical blocks → normalize and score with a learned vector → softmax over the "block" dimension → weighted mix. Note the softmax runs along dim 0 (the block dimension) — the competitors are not tokens, but depths.

13Interlude IV: Game-Changing Optimizations

Architecture is only half the story. What truly carried LLMs from the lab to a billion users over the past seven years is a set of methods that don't change the model's shape but bend the cost curve and the capability ceiling. We organize them along three lines: training, inference, and alignment.

13.1 Training: spend every watt where it counts

Scaling laws and Chinchilla (2020–2022)

OpenAI's 2020 scaling-law paper showed loss falls as a smooth power law in parameters, data, and compute — capability became predictable, giving the industry the confidence to burn money. But DeepMind's 2022 Chinchilla corrected a widespread mistake: models until then were generally "too big, underfed"; the optimal recipe is ~20 tokens per parameter. The 70B Chinchilla crushed the 280B Gopher. The Llama era then swung to "overfeeding" (small models eating trillions of tokens), because inference cost is the long-term dominant expense. This line of thinking persists today: set the inference budget first, then back out the training recipe.

Optimizers: Adam → Muon

AdamW ruled for eight years. In 2024, Keller Jordan's Muon (built for the NanoGPT speedrun) — with its idea of "Newton–Schulz orthogonalized updates for hidden-layer weight matrices" — improved sample efficiency ~2× on multiple benchmarks; Moonshot scaled it to large-model training (Kimi K2 uses MuonClip with QK-clip), bringing Muon into the frontline arsenal. Embeddings and output heads still use Adam — hybrid optimizers are the new normal.

Numerical precision and systems

• BF16 mixed precision (2020+): the training standard; FP16's overflow troubles cured by BF16's large dynamic range.
• FP8 training (DeepSeek-V3, 2024): first validation of FP8 mixed precision at very large scale, with per-block scaling — another cut to training cost.
• Parallelism strategies: the combinatorics of data/tensor/pipeline/expert/sequence parallelism (Megatron, ZeRO, FSDP) — effective compute utilization (MFU) on 10K-GPU clusters grinded from 30% to 50%+.
• Data recipes: dedup, quality filtering, domain mixing, annealing, synthetic data — the industry consensus is "data quality and mix contribute no less than architecture," though every lab's secret sauce stays private.

13.2 Inference: the throughput war

Training spends money once; inference spends it every single day, existentially. The protagonists of this front:

13.3 Alignment and reasoning: RL's second spring

The RLHF era (2022–2023)

The InstructGPT/ChatGPT three-stage recipe — supervised fine-tuning (SFT) → reward model (RM) training → PPO reinforcement learning — turned models from "good at continuing text" into "good at following instructions," the key leap in LLM productization. But PPO is expensive and brittle, so DPO (2023) used a closed-form solution to optimize preference data directly as classification — no reward model, no online RL — and for a time became the open-source community standard.

The reasoning-model era (2024– )

OpenAI's o1 revealed a new scaling dimension: test-time compute — letting the model generate a long chain of thought (CoT) before answering, massively boosting math/code ability. DeepSeek-R1 (2025) open-sourced the path: using GRPO (Group Relative Policy Optimization, no value network) with "verifiable rewards" (is the math answer right, does the code pass tests) for large-scale RL directly — emergent self-reflection and long-chain reasoning. RLVR (Reinforcement Learning with Verifiable Rewards) became the new main course of post-training. Kimi k1.5, Qwen3, and everyone else followed; three curves now run in parallel: pretraining scaling + RL scaling + test-time scaling.

Parameter-efficient fine-tuning

LoRA (2021) freezes the trunk and trains only low-rank bypass matrices, cutting fine-tuning cost by orders of magnitude; QLoRA (2023) adds 4-bit quantization, making single-GPU fine-tuning of a 65B model real. LoRA's idea also flowed back into trunk architectures as "MLA query LoRA" (see Chapter 11) — low rank is one of the hidden themes running through this whole article.

14Timeline 2017–2026

15Conclusion: Beyond Scale

And with that, the journey from GPT-2 to Kimi K3 is complete.

The core change was never scale itself. Every step of architectural evolution changed one of three things: what the model stores, how those states are updated, and how information that a fixed-size state can't hold is retrieved.

Kimi K3 stitches four mechanisms into one system: constant-state recurrent memory (KDA), periodic softmax-precise retrieval (Gated MLA), sparse expert capacity (latent-space MoE), and selective residual access across depth (AttnRes). Every added unit of capacity is spent on something with a definite job.

Finally, for you who started near zero and read all the way here — a minimal knowledge map:

• The skeleton didn't change: embedding → N × (attention + FFN + residual) → LM head. Seven years only touched the organs.
• The main tension: KV Cache (lossless but unbounded) vs fixed state (bounded but lossy). All linearization work approached "fixed state with lossless-retrieval quality," and the final answer was the hybrid.
• Three memory operations: write (outer product) → edit in place (delta rule) → decay (gating); KDA refines decay to per-channel.
• Three efficiency levers: MoE (sparse parameters), MLA (compressed cache), FlashAttention/chunking/WY (IO and parallelism).
• Three capability curves: pretraining scaling, RL scaling (GRPO/RLVR), test-time scaling.

Next time you see "new model X released," try asking three questions: Where does it keep its memory? How does it decide to forget? Where did it add its capacity? — the answer very likely hides somewhere along one of the evolutionary lines this article walked.

AAppendix A: Numerically Stable Softmax & the Online Softmax Proof

A.1 Why subtract the maximum

The problem with computing \(e^{x_i}\) directly: when \(x_i\) is large, \(e^{x_i}\) overflows the FP16/FP32 range (FP16 overflows around \(x > 11\)). Let \(m = \max(x_1, \ldots, x_n)\); the numerically stable Softmax is:

Why equivalent? Divide numerator and denominator by \(e^m\):

After subtracting \(m\), the largest exponent is \(e^0 = 1\) — no overflow; the smallest, \(e^{x_{\min} - m} \approx 0\), doesn't affect the sum.

A.2 Correctness of Online Softmax (proof by induction)

BAppendix B: Deriving the Delta Rule Reparameterization

DeltaNet's recurrent update:

The derivation also reveals a beautiful geometric picture: each delta update is a "controlled reflection" of the state matrix along the direction spanned by the current key — erasing the old projection in that direction and writing the new value. Memory editing turns out to be the composition of geometric transforms.