create-data-product

Creating a New Data Product

CRITICAL OPERATIONAL RULE: You are an end-to-end deployment agent, not just a code generator. Your task is NOT complete until the Quality Gate (Build and Validate) has been executed. You MUST NOT stop after writing the files.

CRITICAL ISOLATION RULE (ZERO-TRUST FOR PLATFORM CODE): When creating or scaffolding a new data product, you MUST not modify or create files in the platform codebase (e.g., cortex-framework-core/src/common/, src/common/). Your edits must be strictly isolated to your custom data product directories (config/<namespace>/, src/data_modules/<namespace>/) and the main configuration file (config/config.yaml). Under no circumstances should you edit common platform utility functions, environment checkers, or services unless explicitly asked to do so by the user.

Step 1: Plan and Confirm

1. Determine SAP Source Version (CRITICAL): You MUST explicitly ask and establish with the user whether the data product is designed for ECC, S/4HANA, or Both. Without this knowledge, proper implementation is impossible.
2. SAP DDIC Metadata Retrieval (CRITICAL): You MUST use the query-sap-ddic skill (via the python3 external-skills/.agents/skills/query_sap_ddic/scripts/query_sap_ddic.py script) to query and retrieve ALL exhaustive details for each involved SAP table directly from the replicated SAP Data Dictionary (DDIC) metadata datasets in BigQuery.
3. Handle Custom SAP Fields (Z-fields, ZZ-fields, YY-fields): Verify if any custom SAP fields are required or exist in the raw replication schema. You MUST use the query-sap-ddic skill to lookup their metadata structure, ABAP technical types, lengths, and check tables, and explicitly document their mapping in your implementation plan.
4. Handle Schema Differences Explicitly: Analyze the retrieved schema specs and explicitly identify and list any structural, field name, or data type differences between ECC and S/4HANA. Your implementation plan must explicitly handle these differences (e.g., using separate folders like annotations/ecc/ vs annotations/s4/, or conditional logic in the SQL definitions).
5. Identify and Validate Foundations: Check available data foundations in src/data_modules/<namespace>/<source>/foundations/sap/ and config/<namespace>/<source>/foundations/sap/. You MUST verify all required tables exist in src/data_modules/<namespace>/<source>/foundations/sap/table_settings.default.yaml. If any tables are missing, explicitly document them in your implementation plan and define a sub-plan to manually scaffold them:
   • Developer Role (extending the codebase): Modify table_settings.default.yaml directly to add the missing tables.
   • End User/Deployer Role (configuring a target deployment): Do NOT modify table_settings.default.yaml directly. Instead, make a copy of table_settings.default.yaml with a custom name (e.g., table_settings.yaml), add the missing tables to it, and reference this new file in the active config.yaml using the tableSettings property. Ensure any new tables are inserted at the correct position to preserve alphabetical sorting by tableName within s4, ecc, and common. Fetch their schema via DDIC lookup skills, and generate their annotations.yaml files in the foundation directory.
6. Verify Custom Namespace: In accordance with the Cortex V7 Extensibility Guide, custom development MUST be placed in a custom namespace (e.g., custom, myorg) rather than the standard cortex namespace to prevent conflicts and ensure clean upgrades.
   • If a custom namespace has not been specified, you MUST explicitly ask the user to provide a custom namespace name before proposing the plan.
   • If the user explicitly instructs you to use the standard cortex namespace, you MUST warn them that modifying the standard namespace is against extensibility best practices and risks being overwritten during future upgrades. Obtain explicit confirmation before proceeding with the cortex namespace.
7. Propose Name: Suggest a descriptive name using snake_case.
8. Legacy Model Reference: If legacy specifications or references are available, check if the data product models an existing or similar legacy data model.
9. Research Field Mappings: Research the relevant field mappings between the data foundation and the new data product models, including the reasoning for each mapping, so the user can adjust them if needed.
10. Evaluate Need for ABAP Source Code (CRITICAL): Depending on the user-provided requirements, your general understanding of SAP systems, the target version (ECC/S4HANA), and the business logic/modeling goals, evaluate if custom or standard ABAP program details are necessary to model the data product correctly (e.g. to replicate complex transaction statuses, calculations, or custom business rules). If required, you MUST explicitly ask the user for the relevant ABAP program names, specifications, or source code blocks.
11. Propose Unit Tests Structure: Formulate the strategy for verifying the product via automated testing by proposing a dedicated Python unit test. Specify the target pytest paths and the relational assertions required to validate the compiled SQL joins, filters, and primary transformations.
12. Propose Plan: You MUST read the template file using your file reading tool at assets/plan_template.md. The plan you propose MUST exactly follow the markdown headings and structure outlined in that file. Do not invent your own structure.
13. WAIT for user confirmation on the plan before writing any code.