classification-at-scale

Classification At Scale

Classify entities from an internal catalog against a standard taxonomy using deterministic providers, relationship-local signal sources, batch group review, and a trust flywheel. LLM reasoning is limited to writing configs, handling ambiguous tails, and critiquing results. The accepted mappings become governed claims in the user's Crux.

When to use this skill

You have two datasets:

• An internal catalog with free-text descriptions and informal categories
• A standard taxonomy with structured types/categories

You need to create classification edges between them — at scale, with receipts, and with minimal ongoing human review.

Architecture boundary

You (the agent):
  - Write the config (entity types, providers, proposal policy)
  - Build or invoke deterministic providers (regex, lookup tables, rules)
  - Run workflows against entities to produce structured extractions
  - Convert extraction results to tri-state signals (support/unsure/contradict)
  - Propose groups through configured workflows or direct agent proposals
  - Run the review loop (sample, critique, refine rules, regroup)

Core:
  - Stores provider identity, contracts, and workflow traces
  - Validates signals against relationship proposal policy
  - Derives review priority from signal + trust state
  - Manages group lifecycle (propose → resolve → trust)
  - Produces receipts for every mutation
  - Gates auto-resolve on Cruxible-computed proposal signatures

Core executes declared providers inside workflows, but it does not own the
domain-specific classification logic. The provider implementation and contract
are the spec of record; relationship `proposal_policy.signals` decides which
signal-source labels govern review.

Phase 1: Understand the data

Before writing any config, profile both datasets.

1. Read headers, row counts, sample rows from both files
2. Identify primary keys (catalog part number, taxonomy type ID)
3. Profile the description fields — look for shorthand patterns, abbreviations
4. Count distinct categories/subcategories on both sides
5. Look for existing classification columns (may be partially populated)
6. Identify junk rows (discontinued, dropbox, test data)