engine

Execution Engine Conventions

• Components implement the Runnable protocol with typed Pydantic I/O schemas.
• _run_without_io_trace(inputs: BaseModel, ctx: dict) -> BaseModel is the core execution method — this is what subclasses implement.
• Port system: PortDefinition (catalogue schema) → PortInstance (per-graph-instance) → FieldExpression (JSON AST wiring and transforms). There is no port_mappings table; field expressions are the sole wiring mechanism.
• GraphRunner uses networkx.DiGraph, topological scheduling, ExecutionDirective for control flow (CONTINUE, HALT, SELECTIVE_EDGE_INDICES). Skipped subgraphs use TaskState.HALTED (root of the skipped branch plus descendants), not COMPLETED. If/Else uses selective edge orders 0 for true and 1 for false/else; the enable_false_path component parameter defaults to false and must be true for the order=1 branch to run.
• Field expressions: 5 AST node types (literal, ref, var, concat, json_build). Key files: engine/field_expressions/ast.py (AST), parser.py, serializer.py, traversal.py.
• parse_expression_flexible() accepts str, dict, and list. Dicts that match a serialized AST shape are deserialized via from_json(). Plain dicts/lists become LiteralNode(json.dumps(value)). The client is responsible for building json_build AST nodes for JSON-typed parameters containing @{{}} refs/vars.
• Variable resolution: defaults → layered set overrides. See ada_backend/services/variable_resolution_service.py.
• Secret handling: runtime secrets use SecretStr (pydantic.SecretStr) and are unwrapped only at explicit execution boundaries (unwrap_secrets() / get_secret_value()).
• Observability boundary: engine/trace/serializer.py masks SecretStr; shared/log_redaction.py is best-effort defense-in-depth for untyped payloads/events and is shared by API, scheduler, engine, and worker processes.
• Default OpenAI chat-capable model is gpt-5-mini; provider-qualified references and completion-model catalog defaults should use openai:gpt-5-mini.
• GoogleProvider must not replay prior assistant tool_calls in multi-turn function-calling requests; Gemini requires hidden thought_signature data that the OpenAI-compatible response does not expose. Preserve tool outputs by converting tool role messages into user-visible tool-result context.
• requires_tool_name = True for components that need a tool_name input (MCP tools, API tools).
• Legacy system: some components still use AgentPayload pattern (migrated = False). New components MUST use the multi-port NodeData system (migrated = True).
• Python Code Runner keeps the full E2B payload in artifacts.execution_result; when a text result exists, it also exposes artifacts.text for @{{instance.artifacts::text}} injection. New sandbox-root files with supported extensions are saved to the run temp folder and exposed as artifacts.files; this key must always be present and contain [] when no supported files were generated. The tool output lists generated filenames so AI Agents can pass them back through input_filepaths on later Python Code Runner calls. When Python Code Runner is an AI Agent tool, AIAgent must carry artifacts.files into the next iteration and add an Available generated files instruction to the system prompt. Generated PDFs must only be attached to the next LLM message after the model explicitly calls attach_generated_files_to_llm with exact filenames; artifacts.attached_files is one-shot metadata and must be cleared after that next request is built. Direct LLM file payload filenames must preserve the normalized run-relative path, not just the basename, so same-named files from different subdirectories remain distinguishable. Skip missing, oversized, or unreadable PDFs per file and keep sending any remaining attachments. If no user-role message is retained, add a fallback user message carrying the file payloads. Its output is trace-friendly and must omit inline png/jpeg base64 payloads while preserving them in artifacts. Shared E2B sandboxes from tracing context must pass a current-loop async health check before reuse; stale or closed-loop clients are discarded and replaced.
• DOCX Template render errors should preserve docxtpl diagnostics: validate undeclared template variables before render, validate image placeholders with python-docx before adding them to render context, identify the failing render substep (body XML, tables, body mapping, headers, footers, properties, footnotes), and include exception type/repr plus Jinja DOCX context when available.
• SQLLocalService engine pools are cached per engine_url (process-level). Avoid patterns that instantiate many services for the same URL expecting independent pools.
• In ingestion code paths, reuse one SQLLocalService per job when possible, and ensure await close() is called in a finally block.
• IS_CLOUD_S3 controls folder-source ingestion URL strategy: keep it False in local/dev and custom S3_ENDPOINT_URL setups (direct file reads; presigned URL getter returns None), set it True in cloud AWS S3 prod to require presigned URL generation and fail fast on missing URLs.
• For DB-source ingestion worker flow (ingestion_script/ingest_db_source.py), keep a single source SQLLocalService per run and share it across validation and fetch helpers.
• Adding a new component: create class in engine/components/, register in ada_backend/services/registry.py, add DB seed data (Component, ComponentVersion, PortDefinition, ComponentParameterDefinition rows). Always set migrated = True, define Pydantic I/O schemas, define canonical ports.
• Branded OAuth components may reuse an existing runtime class when behavior is identical, but must have separate seed entries and a distinct OAuthProvider value when they use a separate Nango provider config key (e.g. google-mail-neverdrop, google-calendar-neverdrop, google-contact-neverdrop). Gmail Neverdrop must remain send-only: no save_as_draft catalog or runtime parameter, recipients required, and no draft fallback. Neverdrop-branded Google tools should remain agent-available (is_agent=True) and function-callable; Google Contacts Neverdrop is read-only, lists regular contacts plus Other contacts by default, always requests People API sync tokens (callers pass sync_token / other_contacts_sync_token for deltas; expired tokens surface EXPIRED_SYNC_TOKEN), exposes contacts_search_contacts over both sources (30-result cap, empty-first-result = cache warmup -> wait ~2s + one retry, restricted Other-contacts search readMask), and requires both contacts.readonly and contacts.other.readonly.
• Unified Mail Sender versions that expose a Gmail Neverdrop connection in seed data must resolve gmail_oauth_connection_id with OAuthProvider.GMAIL_NEVERDROP in ada_backend/services/registry.py; keep older unified Mail Sender versions on regular Gmail when their catalog still exposes OAuthProvider.GMAIL. Standalone Gmail Neverdrop remains send-only with no save_as_draft parameter, while unified mail_sender_v2 exposes save_as_draft for Gmail or Outlook and defaults it to true.
• Mail sender email_attachments is a list whose items can each be either a string path/URL or an object with filename plus either url or path. Preserve both item shapes when changing Gmail, Gmail Neverdrop, Outlook, or unified Mail Sender components; object attachments must use filename as the displayed attachment name when downloading from url or reading from path, and tool-description schemas must not use oneOf/anyOf.
• Outlook Sender cc accepts either a list of emails or a comma-separated string; strings are split/stripped into a list before Graph payload construction.
• Public mail attachment input contract changes need a new ToolDescription row wired only to the new component version; older mail sender versions should keep legacy string-item tool descriptions.
• Attachment URL downloads must validate the initial URL and every redirect target before streaming; reject non-HTTP(S) schemes and private/reserved network addresses. When connecting to a resolved IP for SSRF protection, preserve the original HTTPS hostname for SNI/certificate verification.
• HubSpot MCP crm_upsert_contact_by_email returns id, operation, and remote_url; cache successful portal metadata per HubSpot client and keep remote_url as an empty string if portal metadata cannot be resolved after a successful upsert, including HubSpot metadata API errors and network failures.
• HubSpot MCP notes_upsert_for_contact and tasks_create treat properties.hs_timestamp as optional and auto-fill missing or empty values with the current UTC time in ISO-8601 Z format before calling HubSpot.
• HubSpot Owner is an API-tool-style component, not an MCP wrapper. It keeps HubSpot headers as a JSON configuration value, injects owner_id into https://api.hubapi.com/crm/v3/owners/{owner_id}, and exposes owner fields (id, email, firstName, lastName, etc.) at the root output level for field expressions.
• Google API discovery services used from async MCP clients must be built inside each asyncio.to_thread call; do not share google-api-python-client service instances across worker threads.
• Removing a component: delete the runtime class, registry entry, DB seed definitions, default tool description, and any wrapper components that only exist to invoke it in the same diff. Do not leave dead catalog entries that can still be instantiated through backend graphs or MCP.
• SQLSpanExporter (engine/trace/sql_exporter.py): each span is parsed once (_parse_span_or_error → dict passed to _export_span); do not re-serialize the same span in the export path.
• Hybrid search: QdrantService supports three SearchMode values: semantic (default, dense only), keyword (BM25 sparse only), hybrid (dense + sparse with RRF fusion). All collections are hybrid (named "dense" + "sparse" vectors). Existing component versions default to semantic without exposing the parameter; new versions (RAG v4 0.3.0 in seed_rag_v4.py, Retriever v2 0.0.2, Retriever Tool v2 0.0.2) add search_mode as a user-configurable parameter. See ada_backend/docs/engine.md for details.
• Qdrant metadata filters: keep technical identifiers (chunk_id, source_id, file_id, url, sync_id) as keyword indexes queried with match.value. Never payload-index chunk body fields (content, chunk). Human text metadata (VARCHAR/TEXT names, authors, titles, descriptions) uses Qdrant text indexes and should be queried with match.text so punctuation/case/accent differences are normalized by Qdrant.
• See ada_backend/docs/engine.md and ada_backend/docs/payload-and-data-flow.md.