experiment-log

Experiment Log

Record a training run's parameters, metrics, and artifact paths in the experiments ledger.

Inputs

• Experiment name (e.g., churn-v3-lgbm)
• Run ID (auto-generated if omitted: {name}-{YYYYMMDD-HHmmss})
• Parameters (key-value pairs, e.g., learning_rate=0.05 n_estimators=500)
• Metrics (key-value pairs, e.g., auc=0.83 f1=0.71 precision=0.79 recall=0.64)
• Artifact path (model checkpoint, serialized pipeline, etc.)
• Notes (free-form, optional)

Steps

1. Collect run metadata

   import datetime, hashlib, json, os
   
   run_id = "{name}-" + datetime.datetime.utcnow().strftime("%Y%m%d-%H%M%S")
   git_sha = os.popen("git rev-parse --short HEAD").read().strip()
   dataset_version = open("data/.version").read().strip()  # semver or hash
   

2. Capture parameters and metrics If using a training script that outputs JSON:

   python train.py --config config/{name}.yaml --output-metrics /tmp/metrics.json
   

   Otherwise, capture values directly from the trainer object:

   params = model.get_params()
   metrics = {"auc": roc_auc_score(y_test, y_pred), "f1": f1_score(y_test, y_pred)}
   

3. Save artifacts

   import joblib
   artifact_path = f"artifacts/{run_id}/model.pkl"
   os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
   joblib.dump(model, artifact_path)
   

4. Append to the experiments ledger

   scripts/log-experiment.sh \
     --name "{name}" \
     --run-id "{run_id}" \
     --params '{"learning_rate": 0.05}' \
     --metrics '{"auc": 0.83, "f1": 0.71}' \
     --artifact "{artifact_path}" \
     --notes "{notes}"
   

   The ledger is experiments/log.csv — a flat CSV with one row per run.

5. Verify the entry

   tail -n 5 experiments/log.csv
   

   Confirm: run_id is unique, metrics are numeric, artifact path exists.

6. Compare with previous runs (optional)

   python scripts/compare-runs.py --metric auc --top 5