gradient-detection

Advantage

Gradient Detection can be applied to many types of objects (inputs, activations, parameters) without additional training. It only requires a backward pass, making it efficient for quickly ranking important components. Compared with exhaustive interventions, it can produce rankings with a relatively small number of backward passes, making it practical as an initial localization step when the candidate set is large.

Limitation

Gradients provide a local proxy, not causal necessity: salience can be offset by downstream computation, and finite interventions may depart from first-order effects in non-linear regimes. For these reasons, gradient-ranked objects are typically paired with Causal Attribution to validate whether the identified objects are genuinely responsible for the target behavior.

Submethods

Given object categories, the method typically takes three forms:

• Inputs and Layer-wise States: For input embeddings $x_i^0$ and the residual stream state $x^l$, gradients directly quantify how sensitive $F(x)$ is to changes in specific prompt components and their propagated representations. In practice, one computes $\nabla_{x_i^0} F(x)$ or $\nabla_{x^l} F(x)$ and derives token-level influence, such as the gradient norm $|\nabla_{x_i^0} F(x)|$, the gradient–input score $\nabla_{x_i^0} F(x)^\top x_i^0$, or integrated gradients. Aggregating these scores across positions $i$ (optionally across layers $l$) yields a ranked view of which tokens or contextual spans are most responsible for a target output, as used to analyze CoT prompting and which depth regions contribute most strongly to the formation of that output, with closely related layer-/token-saliency signals also supporting dynamic token pruning and inference-time steering. You can find a demo for this method in ./inputs-and-layer-wise-states.This demo shows layer-gradient: Analyze and visualize layer-wise gradient behaviors in LLMs during fine-tuning for fast vs slow thinking tasks, calculate gradient statistics, and understand training patterns across different model layers