sigma_ingest

Sigma Ingest

Sigma ingest turns staged data model specs and workbook summaries into durable ktx wiki knowledge. The deterministic project() step has already written semantic-layer YAML for all warehouse-table data model elements before this skill runs — do not re-write those SL sources.

Work unit structure

Sigma produces at minimum two work units per ingest run:

• sigma-data-models or sigma-data-models-N
  • rawFiles: data-models/<id>.json files (one per data model in this batch)
  • peerFileIndex: workbooks/<id>.json files + sigma-manifest.json + sigma-projection-config.json
  • When the workspace has more than 50 data models, split into batches: sigma-data-models-0, sigma-data-models-1, … with displayLabel like "Sigma: data models (1/8)". When ≤50 data models, the unitKey is simply sigma-data-models with no suffix.
• sigma-workbooks or sigma-workbooks-N
  • rawFiles: workbooks/<id>.json files (one per workbook in this batch)
  • peerFileIndex: data-models/<id>.json files + sigma-manifest.json + sigma-projection-config.json
  • When the workspace has more than 2000 workbooks, split into batches: sigma-workbooks-0, sigma-workbooks-1, … with displayLabel like "Sigma: workbooks (1/4)". When ≤2000 workbooks, the unitKey is simply sigma-workbooks with no suffix.

sigma-manifest.json and sigma-projection-config.json are never in rawFiles. They live at the staged dir root and always appear in peerFileIndex.

Staged file shapes

data-models/<id>.json — one per data model (in rawFiles for data-model units):

{
  "sigmaId": "abc-123",
  "name": "Revenue Model",
  "path": "Finance/Revenue Model",
  "latestVersion": 3,
  "updatedAt": "2026-01-15T00:00:00Z",
  "isArchived": false,
  "spec": {
    "name": "Revenue Model",
    "pages": [{
      "id": "p1",
      "name": "Main",
      "elements": [{
        "id": "elem1",
        "kind": "table",
        "name": "Opportunities",
        "hidden": false,
        "source": {
          "kind": "warehouse-table",
          "connectionId": "<sigma-internal-uuid>",
          "path": ["DATABASE", "SCHEMA", "OPPORTUNITIES"]
        },
        "columns": [
          { "id": "c1", "name": "Deal Amount", "formula": "[OPPORTUNITIES/Amount]", "description": "Net contract value in USD" },
          { "id": "c2", "name": "Total ARR", "formula": "Sum([OPPORTUNITIES/ARR])", "description": "Annualised recurring revenue" }
        ]
      }]
    }]
  }
}