tokf-filter

tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

Section 1 — What a Filter File Is

A filter file is a TOML file that describes:

• Which command(s) it applies to (command)
• How to transform the raw output (steps, applied in a fixed order)
• What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. .tokf/filters/ — project-local (repo-level overrides)
2. ~/.config/tokf/filters/ — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use tokf which "cargo test" to see which filter would activate for a given command.

Section 2 — Processing Order

Steps execute in this fixed order — do not rearrange them:

1. match_output — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. [[replace]] — per-line regex transforms applied to every line, in array order
3. strip_ansi / trim_lines — per-line cleanup (ANSI stripping, whitespace trimming)
4. skip / keep — line-level filtering (drop or retain lines by regex)
5. dedup / dedup_window — collapse duplicate consecutive lines
6. lua_script — Luau escape hatch; runs after dedup, before JSON/section/parse
7. [json] — JSON extraction via JSONPath; when configured, replaces section/parse/chunk
8. [[section]] OR [parse] — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper). Skipped when [json] is configured.
9. [[chunk]] — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections). Skipped when [json] is configured.
10. Exit-code branch — [on_success] or [on_failure] depending on exit code
11. [fallback] — if neither on_success nor on_failure produced output
12. strip_empty_lines / collapse_empty_lines — post-processing cleanup on the final output