decision-hub: Command for Claude Code

End-to-end test of publishing a skill with evals on the dev environment. Run each step sequentially — stop and report if any step fails.

All CLI commands use: DHUB_ENV=dev uv run --package dhub-cli dhub <command>

Prerequisites

1. DB migration: Before publishing, ensure the dev database has all columns:

   cd server && DHUB_ENV=dev uv run --package decision-hub-server python -c "
   from decision_hub.settings import create_settings
   from decision_hub.infra.database import create_engine, metadata
   settings = create_settings('dev')
   engine = create_engine(settings.database_url)
   metadata.create_all(engine)
   "
   

2. Deploy server: Deploy the latest server code to dev Modal:

   cd server && DHUB_ENV=dev modal deploy modal_app.py
   

Step 1: Create the test skill

Create all files under <scratchpad>/bayesian-ab-test/.

SKILL.md

---
name: bayesian-ab-test
description: "Bayesian A/B testing skill using PyMC for hypothesis testing on log-normally distributed data"
runtime:
  language: python
  entrypoint: run_ab_test.py
evals:
  agent: claude
  judge_model: claude-sonnet-4-5-20250929
---

# Bayesian A/B Test

You are a statistical analysis assistant that performs Bayesian A/B testing.

When asked to run the A/B test, execute the `run_ab_test.py` script and present the results clearly.

## Instructions

1. Run `python run_ab_test.py` to generate the dataset and perform the analysis.
2. Present the output verbatim, ensuring all sections are included:
   - Hypotheses (H0 and H1)
   - Generated data summary
   - Sampling diagnostics (convergence, R-hat, ESS)
   - Posterior summary and credible intervals
   - Conclusion about the null hypothesis
3. Do NOT modify the script or its output.

pyproject.toml

CRITICAL: requires-python MUST be >=3.11 (not >=3.10) because PyMC 5.27.1 requires Python 3.11+. If set to >=3.10, uv sync will fail in the sandbox with a resolution error. Do NOT include a [build-system] section — it causes uv sync to attempt an editable install which fails.