claudini: Instructions file for Claude Code

Claudini — Developer Guide

Package manager

Use uv for everything. uv run -m ... to execute, uv add to install dependencies. Never use pip install.

Running benchmarks

# Run a preset (all methods, all samples)
uv run -m claudini.run_bench random_valid

# Override method, sample, seed, budget from CLI
uv run -m claudini.run_bench random_valid --method gcg,acg --sample 0 --seed 0
uv run -m claudini.run_bench safeguard_valid --method gcg --max-flops 1e15

# Smoke test (use /tmp to avoid polluting results/)
uv run -m claudini.run_bench random_train --method gcg --sample 0 --max-flops 1e12 --results-dir /tmp/smoke

Results save to results/<method>/<preset_name>/<model>/sample_<S>_seed_<N>.json. Existing results are auto-skipped.

Config files

Preset YAML files in configs/ define the full experiment specification:

model: Qwen/Qwen2.5-7B-Instruct
optim_length: 15
max_flops: 1.0e+17
dtype: bfloat16
samples: [0, 1, 2, 3, 4]
seeds: [0]

input_spec:
  source: { type: random, query_len: 0, target_len: 10 }
  layout: { type: suffix }
  init: { type: random }

method_kwargs:        # optional per-method hyperparameter overrides
  acg: { num_coords: 3 }

CLI flags override any preset value.

Adding a new method

1. Create a directory under claudini/methods/ (e.g. claudini/methods/claude/v125/)
2. Add optimizer.py and __init__.py
3. Subclass TokenOptimizer (from claudini.base), set method_name as a class variable:

from claudini.methods.original.gcg import GCGOptimizer

class MyOptimizer(GCGOptimizer):
    method_name = "claude_v125"

    def setup(self, prompt, target):
        super().setup(prompt, target)
        # custom setup

    def step(self, step_num):
        # must call self.flop_counter for every model pass
        self.flop_counter.count_forward(n_tokens)
        # ...
        return (discrete_loss, soft_loss_or_None, suffix_string)

1. In __init__.py: from .optimizer import MyOptimizer
2. Run: uv run -m claudini.run_bench random_train --method claude_v125