Swiss Cheese Validation

Swiss Cheese Validation: Verify Critic Independence

Purpose

Verify that multiple critic layers have orthogonal failure modes, ensuring the Swiss cheese model is working correctly. Measure overlap in critic rejections and warn if critics catch the same errors.

Theory

The Swiss cheese model achieves reliability through misaligned holes:

• Each critic has failure modes (errors it misses)
• Holes don't align across critics
• Errors caught by different critics don't overlap significantly

Target: Independence score > 0.85 (less than 15% overlap)

Validation Process

Measure Overlap

Analyze rejection patterns across sessions:

code_rejections = sessions_where_code_critic_vetoed
security_rejections = sessions_where_security_critic_vetoed

overlap = code_rejections ∩ security_rejections
union = code_rejections ∪ security_rejections

independence_score = 1 - (|overlap| / |union|)

Interpretation:

• Score > 0.90: Excellent independence (orthogonal)
• Score 0.80-0.90: Good independence (acceptable)
• Score < 0.80: Poor independence (critics overlapping)

Expected Results

From research paper (522 sessions):

Critic Pair	Overlap	Independence
Code vs Security	2.3%	0.977 ✓
Code vs Domain	3.1%	0.969 ✓
Security vs Domain	0.8%	0.992 ✓

Warning Indicators

High overlap detected when:

• Same sessions rejected by multiple critics
• Similar rejection reasons across critics
• Independence score < 0.80

Causes:

• Critics using same model/provider (cognitive monoculture)
• Similar training data
• Overlapping validation logic

Solutions:

• Use different model providers per critic
• Adjust critic specializations
• Review critic prompts for duplication

Configuration

Layer Definition

Default (hardcoded):

layers:
  - code-critic
  - security-critic
  - domain-critic

Custom (user-configured):

layers:
  - code-critic
  - security-critic
  - financial-critic
  - performance-critic