A Unifying Lens on SFT Through Target Distribution Design

• 1 Framework. We view the SFT target as a mixture $Q_t = \gamma_t \delta_{y_t} + (1-\gamma_t)\tilde\pi_t$, which relaxes imitation and allows alternatives when the label is uncertain.
• 2 Unify. Many existing SFT variants represent implicit choices of $\gamma_t$ and $\tilde\pi_t$.
• 3 Target-SFT. We design both branches explicitly: $\gamma_t$ via model probability, and $\tilde\pi_t$ via a teacher-guided distribution. It outperforms across all 10 SFT settings evaluated.

Loss is a surrogate. What target does it lead to?

Invert the cross-entropy gradient to find out: $Q_k = p_k - g_k$.

At each prefix $x_t$, the model outputs logits $z$ and a distribution $p = \text{softmax}(z)$. Now suppose there exists some target distribution $Q$ over the vocabulary that we want the model to match. If we train with cross-entropy towards $Q$, then the gradient with respect to a logit $k$ is

This reveals what each loss is actually teaching the model, via the probability updates it defines.

From Loss to Target

Standard SFT learns one-hot target $\delta_{y_t}$; p-loss interpolates from $p$ to $\delta_{y_t}$.

We illustrate this with two losses: standard SFT with negative log-likelihood, and p-loss, a token-level variant scaled by the model's current probability $p_y$. Applying $Q_k = p_k - g_k$ to each derives the target.

The Q-Target View

Relax the one-hot target $\delta_{y_t}$ to account for label uncertainty.

But an observed token may be noisy, non-unique, or misaligned with the model. Instead of forcing a rigid one-hot target, we explicitly model for this uncertainty. We replace $\delta_y$ with a mixture distribution $Q_t$ controlled by two design choices:

The training objective under $Q_t$ decomposes cleanly:

This view shows that SFT training balances two forces: imitation of the label and matching of a residual distribution. Standard SFT simply sets $\gamma_t = 1$, collapsing the second term entirely.

Existing SFT variants are implicit Q-target designs

Seemingly different losses vary only in the choice of $\gamma_t$ and $\tilde\pi_t$.

Design both branches of Q-target

Define proxy for $\gamma_t$ and adaptively use teacher guidance in alternatives $\tilde\pi_t$.

Results

Target-SFT improves reasoning performance across 10 dataset-model settings on math and medical benchmarks. While the baselines fluctuate, Target-SFT consistently achieves the best Average@16 results in every setting.

BibTeX

@article{xie2026targetsft,
  title     = {A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design},
  author    = {Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh},
  journal   = {arXiv},
  year      = {2026}
}