A small model with an unusual spine. Qwen3.5‑4B replaces three‑quarters of its attention layers with a linear‑time recurrent memory — Gated DeltaNet — and keeps only one full‑attention layer in every four. This piece unpacks the architecture from the config up, then traces exactly what happens on a forward pass, with visualizations you can poke at.
Qwen3.5‑4B is a 32‑layer, 2560‑wide, dense decoder. Its layers follow a repeating motif — 3 Gated‑DeltaNet layers, then 1 full‑attention layer, eight times over. The 24 linear layers carry a fixed‑size recurrent state (no growing cache); the 8 full layers do exact softmax attention with grouped‑query heads and partial RoPE. A multi‑token‑prediction head bolts on for faster decoding. That hybrid is the whole reason a 4B model comfortably reaches 262K‑token (and, with YaRN, ~1M) context.
1. The 4B is dense, not MoE. The family’s marketing pairs “Gated DeltaNet” with
“sparse Mixture‑of‑Experts,” but the actual 4B config.json has no expert fields
(mlp_only_layers: [], a single intermediate_size: 9216). Experts live in the
larger siblings; the 4B buys its efficiency from the attention, not from routing.
2. This checkpoint is multimodal (vision‑language). The config carries a
vision_config, image_token_id and M‑RoPE. Everything below describes the shared
text backbone — which is where all the interesting architecture lives. If you actually want the
audio / Omni variant, its config differs at the encoder, not the decoder spine.
Everything here is read straight from the published config.json, not from memory. The headline
numbers:
| Config field | value | What it controls |
|---|---|---|
| num_hidden_layers | 32 | Decoder blocks |
| layer_types | [lin×3, full]×8 | linear full repeating motif |
| full_attention_interval | 4 | Every 4th layer is full attention |
| hidden_size | 2560 | Residual stream width |
| num_attention_heads | 16 | Query heads (full layers) |
| num_key_value_heads | 4 | KV heads → GQA 4:1 |
| head_dim | 256 | Per‑head width (Q proj is 16×256=4096) |
| partial_rotary_factor | 0.25 | RoPE on 64 of 256 head dims |
| attn_output_gate | true | Sigmoid gate on attention output |
| linear_num_value_heads | 32 | Gated DeltaNet V heads |
| linear_num_key_heads | 16 | Gated DeltaNet Q/K heads |
| linear_{key,value}_head_dim | 128 | DeltaNet head width |
| linear_conv_kernel_dim | 4 | Short causal conv before the recurrence |
| intermediate_size | 9216 | SwiGLU inner width |
| hidden_act | silu | Activation → SwiGLU |
| vocab_size | 248320 | Tied input/output embedding |
| rope_theta | 10000000 | RoPE base frequency |
| rms_norm_eps | 1e-6 | RMSNorm epsilon |
| mtp_num_hidden_layers | 1 | Multi‑token‑prediction head |
A plain transformer stacks 32 identical blocks. Qwen3.5 instead repeats a 4‑layer motif eight
times. Three cheap linear‑attention layers carry a constant‑size recurrent memory; then one true
softmax‑attention layer every fourth block does exact long‑range lookups. 75% of the token‑mixing is
O(n) with fixed memory, 25% is O(n²) for precision.
Click any layer below to see what it is and what it costs.
Each block = token‑mixer (DeltaNet or attention) → SwiGLU FFN, both pre‑normed with residuals. The pattern is DeltaNet · DeltaNet · DeltaNet · Attention, ×8.
Zooming into a single repeat of the motif. Both layer flavors share the same skeleton — pre‑norm, token‑mixer, residual, pre‑norm, SwiGLU FFN, residual — and differ only in the mixer.
This is the layer that makes long context cheap. Instead of storing every past key/value in a cache that
grows with sequence length, a DeltaNet layer keeps a single fixed‑size state matrix
S per head and updates it token by token — a linear RNN. Think of S as an
associative key→value memory: S · k retrieves the value currently bound to key k.
If you already picture self‑attention as “each token queries the others, gets weights from query·key, and adds their weighted values back,” then DeltaNet is that exact story, algebraically rearranged. Everything below falls out of one question: what if we delete the softmax? Per‑head dimensions:
dk = key/query dim = 128 · dv = value dim = 128
· T = sequence length (e.g. 10). A plain vector like qt is a column
(128×1); its transpose qtᵀ is a row (1×128).
0 · Your transformer picture, in one line. Token t scores its query against every key, softmaxes into weights, and takes the weighted sum of values:
Everything about DeltaNet falls out of asking one thing about this: what if we delete the softmax?
1 · The softmax is the only thing forcing O(n²). Attention costs O(n²) because softmax(qt · ki) is nonlinear and couples qt and ki inside an exp. You can't separate the q part from the k part, so for every token you must revisit every earlier key and recompute the pairwise score — that's the n×n score matrix.
So drop it. Replace the softmax weight with the raw dot product qt · ki (this is “linear attention” — you can also wrap q,k in a feature map, but identity is enough to see the idea):
Same idea — a weighted sum of values, weights from query·key — just without the exp and the normalization. Each weight is a scalar, each vi is a 128×1 column, so ot is a 128×1 column: the shape of one value vector.
2 · The one algebra trick that creates the “state matrix”. Look at a single term (qt · ki) vi. The dot product is just a scalar, so slide it around and rewrite it as an outer product times a vector:
That middle step is the whole magic — just associativity, (q·k)v = (vkᵀ)q. The key thing for the dimension: vi kiᵀ is an outer product — a (128×1) column times a (1×128) row — so it's a 128×128 matrix, not a vector. Now sum over all past tokens and pull qt out of the sum:
This is the punchline. Instead of comparing qt against a growing list of keys, you keep a single running matrix, and reading is one matrix–vector product:
That recurrence — “each token adds its outer product vt ktᵀ to the running matrix” — is
the entire reason there's a state matrix. It's not a new mechanism; it's your attention sum, reorganized so you
never build the n×n table.
St is dv × dk = 128 × 128. Read it as a table with
128 columns (one address‑slot per key dimension) and 128 rows (the value stored at each slot).
Writing is v kᵀ = a (128×1)(1×128) = 128×128 stamp; reading is S k =
(128×128)(128×1) = 128×1, a value. Because Qwen sets dk = dv = 128, S
is square — but conceptually it's always “value‑dim by key‑dim.” Crucially it never depends on T:
10 tokens or 10 million, S stays 128×128.
And it's still literally the weighted sum of values. Expand St qt back out:
Same “weighted sum of other tokens' values, weighted by key·query match” you know from attention. The only thing
lost versus softmax is the normalization/sharpening that exp and the denominator gave you. DeltaNet
doesn't restore a softmax denominator; it controls scale with the gating below plus a final RMSNorm.
3 · Why plain accumulation is dumb → the delta rule. St = St−1 + vtktᵀ has two flaws: it only ever adds (reuse a key and you get vold + vnew blurred together), and it never forgets (S grows unbounded and saturates). Fix #1 is the delta rule: before writing, read what's already stored at kt, measure the error, and correct it:
The error (vt − St−1kt) is 128×1; times ktᵀ (1×128) gives a 128×128 correction. Multiply out and regroup:
The (I − βt kt ktᵀ) term erases whatever was stored at kt first (I is 128×128), then you stamp the new value. This isn't a hack — (vt − St−1kt)ktᵀ is exactly the gradient of ½‖S kt − vt‖², so each token does one step of online gradient descent (the Widrow–Hoff delta rule). Fix #2 is the gate: decay old state by αt ∈ (0,1) so stale memory fades:
αt = learned forget gate, βt = learned write strength (both scalars per head, per token). “Gated” + “delta rule” = Gated DeltaNet — just smarter versions of S += v kᵀ.
4 · The correspondence, side by side (same 10 tokens, one head):
| Step | Softmax attention (what you know) | DeltaNet (linear) |
|---|---|---|
| per token | q,k,v ∈ ℝ¹²⁸ | q,k,v ∈ ℝ¹²⁸ (+ scalars α, β) |
| the “context” object | scores Q Kᵀ = (10, 10)/head | state S = 128×128 /head |
| object size vs seq len | grows (n×n) | fixed (128×128, independent of 10) |
| read for token t | softmax(row t) · V | St · qt |
| how token t gets context | scores each earlier token individually | queries S, a running blend of all earlier tokens |
| update as a token arrives | append k, v to the cache | one 128×128 update to S |
| residual | xt + Wo·ot | xt + Wo·(σ(gate) ⊙ ot) |
The translation of “each token queries the others and they answer with their keys”: in DeltaNet, every earlier
token has already deposited its vi kiᵀ into S, so when
qt queries S it queries all of them at once — the delta rule just makes sure each
deposit cleanly overwrites its own address instead of smearing. The equation below is exactly what we just derived.
For each token the state updates as:
kt currently points to, so you overwrite instead of pile on. It's exactly one step of online gradient descent on ‖S kt − vt‖².vt to key kt with strength βt.
That erase‑then‑write is why DeltaNet retrieves in‑context facts far better than vanilla linear attention,
which just accumulates and blurs. A short causal conv (kernel 4) mixes each token with its 3
predecessors before the recurrence, and the output gate σ(g) is what keeps
activations numerically sane.
The equation above hides a very simple picture. A DeltaNet head's memory is one matrix S — draw it
as a grid of colored cells. Everything the layer does is two moves on that grid: writing (an
outer product) and reading (a matrix–vector product). Think of S as a wall of
pigeonholes: the key picks a pigeonhole, the value is what you drop in.
v ⊗ key kᵀ = a stamp into one column of SS becomes a sum of these stamps — that's the whole memory.S × query q = the addressed column, pulled back outo = S·q. A one‑hot query pulls out exactly one column — the value stored there. A fuzzy query pulls a blend of columns, weighted by how well each stored key matches it. That weighted blend of stored values is attention — just pre‑baked into S instead of recomputed over a cache.Now the full story on a 4‑slot memory. Two tokens write, we read one back, then a third token reuses a slot — and you can see the delta rule erase before it writes, which is the one thing plain linear attention gets wrong. Step through it:
S is a live key→value memory, and the delta rule keeps it clean by wiping an address before overwriting it.Same idea, now with sliders and live numbers. Each column of the heatmap is one memory slot (one key); each row is a value dimension. Write a value to a slot, then query it. Turn write strength β down and writes blur together; turn decay α down and old memories fade. Write to the same slot twice to watch the delta rule cleanly overwrite.
i replaces column i with (1−β)·old + β·new; the decay tick multiplies the whole state by α. That per‑column overwrite is the delta rule.
Figure 3 gave you the intuition; now the exact machinery. Two parts: first the tensor shapes
for one real DeltaNet layer (so you see how S fits into the 2560‑wide stream), then a
3‑token walkthrough of a single head with real numbers.
Every shape below is forced by the config: hidden 2560, 16 key heads × 128,
32 value heads × 128, conv kernel 4. Shapes are per token; in prefill you
stack T of them.
| # | Tensor | shape | where it comes from |
|---|---|---|---|
| 1 | input xt | (2560,) | the residual stream entering the layer |
| 2 | RMSNorm(x) | (2560,) | zero‑centered pre‑norm |
| 3 | q = x·Wq | (2048,) → (16, 128) | 16 key/query heads × 128 |
| 4 | k = x·Wk | (2048,) → (16, 128) | same head count as q |
| 5 | v = x·Wv | (4096,) → (32, 128) | 32 value heads × 128 |
| 6 | β (write) = σ(x·Wβ) | (32,) | one write‑strength scalar per value head |
| 7 | α (decay) = f(x·Wα) | (32,) | one forget gate in (0,1) per value head |
| 8 | gate = x·Wg | (4096,) → (32, 128) | the output gate σ(gate) |
| 9 | causal conv1d(k=4) on q,k,v | shapes unchanged | mixes each token with its 3 predecessors |
| 10 | L2‑norm q,k; expand 16→32 heads | (32, 128) | GQA: each q/k head serves 2 value heads |
| 11 | state St (per head) | (128, 128) × 32 | the delta‑rule recurrence updates this |
| 12 | read‑out o = S·q | (32, 128) → (4096,) | retrieve from each head's memory |
| 13 | o · σ(gate), RMSNorm | (4096,) | gated + normalized output |
| 14 | out = o·Wo | (4096,) → (2560,) | project back to the stream width |
| 15 | xt + out | (2560,) | residual add → next layer |
A full‑attention layer caches every past K,V — it grows with context. A DeltaNet layer's entire history is
just its 32 state matrices (128×128 each) plus a 3‑token conv window. That's a
fixed ≈1 MB per layer, regardless of whether you're at token 4 or 400,000. In decode you overwrite
S in place — you never grow anything. That single fact is the whole efficiency story of the
linear layers.
Real heads use d = 128; here we shrink to d = 2 so the numbers fit on screen. One
head, so the state S is a single 2×2 matrix — the same recurrence runs 32× in
parallel in the real layer. Think of the two key dimensions as two memory slots,
A = [1,0] and B = [0,1]. Step through and watch the delta rule
write, coexist, and overwrite.
St = α·St‑1(I − β kₜkₜᵀ) + β vₜkₜᵀ · read‑out o = S·q(I − β kkᵀ) zeroes slot A first, so the read‑out jumps cleanly from [2,0] to [5,0]. Meanwhile α = 0.9 quietly fades slot B from 3 → 2.7.Every fourth layer is ordinary causal softmax attention, but with the Qwen3‑era refinements. These 8 layers are the only place the model does exact all‑pairs lookups — and the only place it keeps a growing KV cache.
partial_rotary_factor 0.25); the rest are position‑agnostic, which helps length extrapolation. On this VL checkpoint those 64 dims use M‑RoPE split [11, 11, 10] across text/height/width.σ(g) gating as the DeltaNet layers, applied to the attention output.Every layer's second sub‑block is a gated MLP:
Standard SwiGLU — one branch gates the other. Nothing exotic; it's the same everywhere.
mtp_num_hidden_layers: 1 adds a small extra block trained to predict the next‑next token.
At inference it acts as a built‑in speculative‑decoding draft head: propose 2 tokens, verify them in one main
forward pass, and roughly double decode throughput when the guess is accepted. It's a training‑time addition
that pays off at serving time.
Inference has two regimes, and the hybrid design behaves very differently in each. Toggle between them:
[T, 2560] residual stream.S per head. No KV entries stored.In a pure 4B transformer, every decode step re‑attends over a KV cache that grows linearly in all 32 layers. Here, 24 layers hold a fixed state and only 8 keep a cache. Drag the context length and watch the KV footprint diverge (bf16, this config's exact head counts):
Qwen3.5‑4B's design is one bet stated three ways: most layers don't need to look at every past token exactly. Compress history into a fixed‑size, error‑correcting memory (Gated DeltaNet) for 24 layers; spend exact attention only where it counts (8 layers); and add a draft head (MTP) to speed up generation. That's how a 4B dense model serves million‑token context at a fraction of a plain transformer's cache cost.
Let's make it fully concrete. Input string Capital of France, assume one token per word, and the
model should continue is Paris then stop. We'll trace every matrix and its dimensions
through the first DeltaNet layer and the first full‑attention layer — first in prefill (all 3
tokens at once), then in decode (feeding the single new token is). Use the toggle;
every shape in both tables updates so you can see exactly what changes.
The 3 tokens become 3 IDs, each looked up in the 248320 × 2560 embedding table:
That (3, 2560) tensor — 3 rows, one 2560‑vector per token — is what climbs the 32 layers. In
decode it will instead be (1, 2560): just the new token. Everything below is one layer;
the real model repeats the motif (DeltaNet ×3, attention ×1) eight times.
T = number of tokens flowing in (3 in prefill, 1 in decode). Weight shapes are fixed; only the
leading token dimension changes. Highlighted rows are where prefill and decode genuinely differ.
| # | Operation | shape | what & why |
|---|
The projections, conv, gate and output are identical in both modes — same weights, just a different
leading dimension (3 vs 1). The only real difference is the recurrence: prefill folds 3 tokens
into S from zero (done as a chunked parallel scan); decode loads the saved S,
applies one update, and overwrites it in place. No past tokens are re‑read, because the past is
S. Cost per decode step is constant — this is why the 24 linear layers barely notice context length.
(For the actual arithmetic of one such update, see Figure 3B above.)
Same input tensor, very different mixer. Note the asymmetry: 16 query heads but only 4 KV heads
(GQA 4:1), each head 256‑wide, and RoPE applied to just 64 of those 256 dims.
| # | Operation | shape | what & why |
|---|
Here the past is not compressed — it's kept verbatim in the KV cache. Prefill computes
a full 3×3 causal score matrix and stores 3 cache entries. Decode appends the single new token's
K,V (cache 3 → 4) and computes a 1×4 score row: the new token is
attends over all cached keys — including France, which is exactly the exact‑lookup the model
needs to answer Paris. This cache grows every step, so its cost scales with context — but it's only
8 of the 32 layers.
After either mixer, the same tail runs: residual add, then a SwiGLU FFN (Wgate, Wup:
2560→9216, Wdown: 9216→2560), then another residual — output shape unchanged at
(T, 2560). That feeds the next layer. After all 32 layers, take the last token's
(2560,) vector → final RMSNorm → LM head 2560 → 248320 → softmax over the vocabulary:
France's vector; its logits peak on is. That's output token #1. Carried forward: 8 KV caches (3 entries each ≈ 96 KB) + 24 DeltaNet states (fixed ≈ 24 MB).is → its logits peak on Paris. Cache grows to 4 entries.Paris → logits peak on <EOS>. Generation stops.
Three tokens generated (is, Paris, <EOS>) in three forward passes:
one prefill (which itself emits is) plus two decodes.