mechanism-behavior-discovery

Mechanism — Behavior Discovery

The discovery half of the loop: before you can explain why a model does something, you need a behavior worth explaining. This skill helps surface a new behavioral phenomenon — a candidate claim — and choose the data that tests it. The sharpened phenomenon (a one-sentence falsifiable behavior, its data/metric, and a plausible internal locus) hands off to /mechanism-explore.

A phenomenon is an observable, reproducible regularity in a model's input→output behavior that is not obvious a priori. A candidate is worth pursuing when it is real, non-obvious, specific, robust, and tractable (a plausible internal locus exists to explain it).

When to Use

The task is open-ended — "find something interesting about how this model behaves," "what's surprising here." Do not use it to explain an already-named behavior (that is /mechanism-explore) or to score a model on a fixed benchmark.

This skill runs only when the phenomenon is not already pinned by the user. When the user explicitly names the phenomenon to investigate, the caller skips discovery entirely and goes straight to explaining that named phenomenon — so a behavior-level override is handled by the caller, not here.

Strategies for Choosing a Behavior to Investigate

1. Transfer a behavioral phenomenon into a high-stakes domain. Take a behavior already known elsewhere, move it into an important domain, and test whether it reappears — either under the same conditions or under stricter, more counterintuitive ones. Domains include but are not limited to:
   • Science domains: chemistry, biology, medicine, …
   • Language: how language systems evolve and develop, etymological / cognate relationships, ancient-text decipherment, and the language–intelligence relationship.
   • Multi-agent social science.
   • Creativity.
2. Borrow from the human sciences.
   • Take a finding from brain science, psychology, or developmental history and check whether LLMs exhibit the same behavior.
   • Compare how the human brain and LLMs process the same task, identifying similarities and differences. This usually requires EEG (or other neural) recordings of humans performing that task.
3. Cross-modal transfer. Take a phenomenon seen in text and check whether it appears in image / video / multimodal models.
4. Reuse existing results in computer science. Check whether earlier findings, methods, or conclusions in computer science apply to the current model or research question.
5. Probe a phenomenon's conditions or causal origin. Take a known (or just-surfaced) phenomenon and ask when it holds or why it arises.
   • When it holds — characterize the regime of validity. Macro: under what general condition or law does the phenomenon hold or break? Micro: vary a concrete knob — model scale, checkpoint, prompt format, language, in-context examples, difficulty, or domain — and find the specific point at which the behavior flips. Either a general boundary or a single flipping condition is itself a candidate claim.
   • Why it arises — trace it to a training cause (data frequency, order of acquisition across checkpoints, objective, RLHF stage) or an inference cause (decoding, attention/representation locus, prompt position, context length), yielding a claim of the form "P is caused by C at stage S".
6. Meta-analysis. Distill a theory or law from prior research — e.g. the scaling law and the Densing Law of LLMs — and use a macro-level or mathematical-theory lens to characterize the regularity, including the conditions under which a given phenomenon holds.