A phase diagram for training and inference in language models

语言模型的每一层,
都是相图上的一个点。

Every layer of a language model
is a point on a phase diagram.

训练后的模型停在中央的工作相;未训练的网络大多漂向坍缩或噪声。训练把推理流拉进工作相——我们把它解释为一次相变。

Trained models stay in a central working phase; most untrained networks drift toward collapse or noise. Training pulls the inference flow into the working phase — which we interpret as a phase transition.

Phase diagram schematic Schematic: a red inference flow settles at the star in the central working phase; two blue flows pass by and drift away, one to the collapse corner and one to the noise corner. u dC 工作相working phase 噪声区noise 坍缩区collapse

01 / 定义Definition

一层的表示,
用两个平行的坐标描述。

Describe one layer's representation
with two parallel coordinates.

沿着残差流,Transformer 是一个沿深度演化的离散动力系统。在某一层,一篇文档的全部 token 表示是高维空间里的一团点云。我们对它只问两个互相独立的问题:文档之间区别有多大,文档本身有多复杂。

Through its residual stream, a Transformer is a discrete dynamical system along depth. At one layer, all token representations of a document form a cloud in a high-dimensional space. We ask it two independent questions only: how different documents are from one another, and how complex each document is in itself.

TOKEN CLOUD一篇文档 = 一团点云A document = a cloud

$\delta$:这篇文档的均值相对其他文档均值(十字)的偏移;$C$:这篇文档 token 的协方差(虚线椭圆),刻画点云的形状。$\delta$: the offset of this document's mean from the mean of the other documents (cross); $C$: the covariance of this document's tokens (dashed ellipse), the shape of the cloud.

$u$ · DISTINGUISHABILITY文档之间区别有多大How different documents are$u = \dfrac{|\delta|^2}{\sqrt{\operatorname{tr} C^2}}$

文档均值的偏移 $\delta$,相对于文档内的散布。$u$ 越大,越容易仅凭一个 token 认出它来自哪篇文档。The offset $\delta$ of the document mean, relative to the spread within. The larger $u$, the easier it is to tell a token's document from that token alone.

$d_C$ · DIMENSION文档本身有多复杂How complex a document is$d_C = \dfrac{(\operatorname{tr} C)^2}{\operatorname{tr} C^2}$

文档内的变化有效地铺在多少个方向上:均匀铺在 $k$ 个方向时 $d_C = k$,介于 1 与嵌入维数 $D$ 之间。Over how many directions the variation within the document spreads: evenly over $k$ directions gives $d_C = k$, between 1 and the embedding dimension $D$.

8 个训练后模型 · 96 篇长文档8 trained models · 96 long documents

两条深度曲线合起来,就是平面上的一条轨迹Two depth curves make one trajectory on the plane

1.00
左:$u$ 随深度先快后慢地上升,$d_C$ 先略升、后缓降;细线是各个模型,红线是中位数。右:把同一深度的 ($u$, $d_C$) 画成一个点,两条曲线就合成平面上的一条轨迹。8 条轨迹在深层聚到一起——下一节会看到,这里就是工作相。一个模型在全部文档上的轨迹,就是它的推理流。Left: $u$ rises fast, then slowly, with depth; $d_C$ rises slightly, then declines; thin lines are the models, the red line their median. Right: plotting ($u$, $d_C$) at the same depth as one point turns the two curves into one trajectory on the plane. The 8 trajectories gather in deep layers — the next section shows this is the working phase. A model's trajectories over all documents make up its inference flow.

02 / 相图Phase diagram

工作相在中央,
漂移相通向四个极限。

The working phase sits in the middle;
the drifting phase leads to four limits.

把训练后模型与未训练网络深层的状态放在一起,($u$, $d_C$) 平面分成两个区域:训练后模型停留的工作相,和未训练网络漂向的漂移相。漂移相通向四个极限,最常见的是左上的噪声区和右下的坍缩区。

Pool the deep-layer states of trained models and untrained networks, and the ($u$, $d_C$) plane splits into two regions: the working phase where trained models stay, and the drifting phase toward which untrained networks drift. The drifting phase leads to four limits; the most common are the noise region at top left and the collapse region at bottom right.

同样三个问题,只有工作相正常回答Three questions: only the working phase answers normally

示意。噪声区($u$ 小、$d_C$ 大)里文档之间分不开,不同的问题得到同一种回答;坍缩区($u$ 大、$d_C$ 小)里每篇文档只剩一两个方向,回答陷入重复;工作相两者兼顾,回答正常。Schematic. In the noise region (small $u$, large $d_C$) documents cannot be told apart, so different questions get the same answer; in the collapse region (large $u$, small $d_C$) each document keeps only one or two directions, so answers fall into repetition; the working phase keeps both and answers normally.

拖动圆点,看点云怎么变Drag the dot and watch the clouds change

左:实测相图,拖动圆点或点上方的按钮。右:同一位置上四篇文档的点云(示意,一种颜色一篇文档)——$u$ 越大,点云彼此离得越远;$d_C$ 越大,每团点云越圆。Left: the measured diagram — drag the dot or press a button above. Right: the token clouds of four documents at that position (schematic, one colour per document) — the larger $u$, the farther apart the clouds; the larger $d_C$, the rounder each cloud.
实测 · 156 个未训练网络 · 8 个训练后模型Measured · 156 untrained runs · 8 trained models

推理流停在哪里,决定它的相Where an inference flow ends decides its phase

训练后 · 在工作相内trained · in the working phase–
未训练 · 在工作相内untrained · in the working phase–
每条线是一个网络的文档中位轨迹,由浅到深是从第一层到最后一层;悬停可查看单个网络。按下播放,所有网络按相对深度逐层推进,下方实时统计此刻有多少网络在工作相内。训练后模型在相对深度约 0.11 处进入工作相,此后每层都留在里面。未训练网络中,小初始化尺度的大多走向坍缩区,大尺度的走向噪声区,与随机 Transformer 中的秩坍缩和熵坍缩一致。Each line is one network's document-median trajectory, light to dark from the first to the last layer; hover to see a single network. Press play and all networks advance layer by layer in relative depth, with a live count of how many are in the working phase. Trained models enter the working phase at relative depth about 0.11 and stay at every later layer. Among untrained networks, small initialization scales mostly lead to the collapse region and large ones to the noise region — consistent with rank collapse and entropy collapse in random Transformers.

8 个训练后模型来自 5 个家族、4 家机构,全部进入并停在工作相;它们深层位置的离散度只有 0.085 dex,未训练组为 0.3–1.4 dex。The 8 trained models — 5 families, 4 institutions — all enter and stay in the working phase; their deep-layer positions scatter by only 0.085 dex, against 0.3–1.4 dex for untrained runs.

03 / 动力学Dynamics

运动趋势随状态而变:
一个差分方程决定多层的行为。

The tendency to move depends on the state:
one difference equation sets the multilayer behavior.

只看位置分不清两种情况:有些未训练的推理流也会穿过工作相,只是不停留。所以我们把逐层的位移写成状态的函数——下一层怎么走,由这一层在哪里、上一层怎么走决定。

Position alone cannot tell two cases apart: some untrained flows also pass through the working phase, without staying. So we write the layer-to-layer step as a function of the state — how the next layer moves is set by where this layer is and how the previous one moved.

$v_{t+1} = \alpha\, v_t + F(x_t) + \eta_t$
$\alpha$惯性:上一层的位移有多少带到下一层inertia: how much of the last step carries over$F$有效力:只由当前位置决定的平均位移effective force: the mean step set by the current position$\eta$涨落:其余自由度留下的无偏扰动fluctuation: the unbiased remainder

$x_t$:第 $t$ 层在相图上的位置;$v_t = x_t - x_{t-1}$$x_t$: the position of layer $t$ on the diagram; $v_t = x_t - x_{t-1}$

t = 0
放大 $P^*$ 附近。灰色箭头是实测的有效力;黑色箭头是 $P^*$ 处的两个特征方向,指向内为恢复、向外为发散。按下播放,圆圈上的 16 个状态按 $P^*$ 附近线性化的方程逐层前进(略去涨落)。Zoom around $P^*$. Grey arrows are the measured effective force; black arrows are the two eigendirections at $P^*$, inward restoring and outward diverging. Press play and the 16 states on the ring advance layer by layer under the equation linearized near $P^*$ (fluctuation dropped).

04 / 训练Training

训练改写差分方程,
把推理流拉进工作相。

Training rewrites the difference equation
and pulls the inference flow into the working phase.

以训练步数为控制参数,在 Pythia-1.4B 的 19 个检查点上测量同一个网络。按下播放,看深层状态怎样随训练进入工作相、惯性怎样随之改变。

With the training step as the control parameter, we measure the same network at 19 checkpoints of Pythia-1.4B. Press play to watch the deep-layer states enter the working phase and the inertia change with them.

第 256 到 512 步之间,推理流进入工作相Between steps 256 and 512 the flow enters the working phase

step 0
左:大圆点是深层(后半段各层)状态的中位点,黑线是它在各检查点走过的路径,灰点是 96 篇文档各自的位置。右:工作相占比 $\varphi$、惯性 $\alpha$ 和收缩率 $-\ln\rho$,竖虚线是两相的分界,左侧小框是第 0 步。第 256 到 512 步之间,$\varphi$ 从几乎为零跳到绝大多数,$\alpha$ 同时骤降到 0 附近;收缩率进入工作相后转为正,但没有跳变。第 0 步的深层状态也在工作相里,惯性却是漂移相的值:推理流只是路过。Left: the large dot is the median of the deep-layer (late-half) states, the black line its path over the checkpoints, and grey dots the 96 documents. Right: the working fraction $\varphi$, the inertia $\alpha$ and the contraction rate $-\ln\rho$; the dashed line is the phase boundary and the small box on the left is step 0. Between steps 256 and 512, $\varphi$ jumps from almost zero to the vast majority and $\alpha$ drops to near 0; the contraction rate turns positive in the working phase, without a jump. At step 0 the deep-layer states also sit in the working phase, yet the inertia has its drifting value: the flow only passes through.
解读READING

工作相占比跳变的同时,惯性骤降到 0 附近——按两相比例加权(杠杆定则)即可再现。我们把这次转变解释为一次相变,与训练中已报道的突变一致,例如 grokking 中泛化的出现、induction head 的形成。As the working fraction jumps, the inertia drops to near 0 — reproduced by weighting the two phases by their fractions (the lever rule). We interpret this as a phase transition, consistent with abrupt changes reported during training, such as the onset of generalization in grokking and the formation of induction heads.

05 / 讨论Discussion

已有的观察,
成为同一张图上的位置与变化。

Existing observations
become positions and changes on one map.

相图用到的量和视角来自许多已有工作,相图上观察到的现象又把许多已有结果连在一起。下表把它们与本文对照,可下载为 CSV 或 BibTeX。

The quantities and perspectives of the phase diagram come from many existing works, and the phenomena seen on it connect many existing results. The table sets them against this paper and can be downloaded as CSV or BibTeX.

相图字典Phase-diagram dictionary