collective-adjudication

Collective Adjudication

Purpose

Aggregate rankings from multiple independent judges into a single consensus ranking. Handles disagreement detection, voting paradoxes, and produces transparent aggregation with disagreement maps.

When to use

• Multiple judges/evaluators available (≥3)
• LLM-as-judge with multiple prompting perspectives
• Committee decision-making requiring formal aggregation
• Need to identify and characterize disagreement patterns

Budget

Resource	Allocation
Judges/Perspectives	≥3 independent evaluators
Comparisons per judge	Complete or near-complete per judge
Aggregation methods	≥2 methods for robustness check
Disagreement threshold	Flag pairs where judges disagree >40%

State Ledger

candidates: []
perspectives: []        # judge identities/prompts
ballots: []             # [{judge, ranking: [...]}]
aggregation_results: {} # method → consensus_ranking
disagreement_map: {}    # pair → {agreement_rate, split}
cycles: []              # Condorcet cycles if any
method: ""              # schulze | borda | kemeny-young | copeland

Available Tactics

• multi-judge-aggregation — collect ballots, aggregate, identify disagreement
• consistency-audit-loop — detect cycles in aggregated preferences

Available SOPs

• ballot-collection
• aggregation-method
• cycle-detection
• inconsistency-localization
• ranking-synthesis

Execution Guidance

1. Define perspectives (judge roles, prompting strategies)
2. Run ballot-collection to gather independent rankings
3. Run aggregation-method with primary method (Schulze recommended)
4. Run cycle-detection on aggregated pairwise matrix
5. If cycles exist, run inconsistency-localization
6. Cross-validate with secondary method (Borda or Copeland)
7. Produce final ranking with disagreement heatmap

Output Format

consensus_ranking:
  - {rank: 1, candidate: "...", wins: 8, copeland_score: 0.95}
  - {rank: 2, candidate: "...", wins: 7, copeland_score: 0.88}
method: schulze
judges: 5
condorcet_winner: "candidate_a"  # or null if cycle
disagreement_hotspots:
  - {pair: ["c", "d"], agreement: 0.4, split: "3:2"}
cross_validation: {borda_agreement: 0.92, copeland_agreement: 0.96}