add-middleware

Add a Middleware Type

Companion Guidance

Use karpathy-guidelines alongside this skill for implementation or review work. Keep changes scoped, surface assumptions, and define focused validation before editing.

NeMo Relay supports guardrails (validate/gate) and intercepts (transform) at various pipeline stages. Adding a new middleware type requires changes across all layers.

Use this skill when introducing a new middleware registration surface or adding middleware behavior to a new pipeline stage.

Lock The Design First

Decide these before editing code:

• Is this for tools, LLMs, marks, scope events, or a combination?
• Is it a conditional guardrail, sanitize guardrail, request intercept, or execution intercept?
• Does it run on request input, inner callable execution, stream chunks, or final response output?
• Is the callback fallible, and how should callback failures propagate?
• Does it need both global and scope-local registration?
• What should subscribers and exporters observe in the event payload after this middleware runs?
• If this is an event sanitizer, which of data, category_profile, and metadata can change, and is the event used only as immutable context?

Pipeline Order

Refer to docs/about-nemo-relay/concepts/middleware.mdx for the full diagrams.

• Tool execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
• LLM execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
• Mark and scope events: specialized tool or LLM sanitizer (when applicable) -> mark or scope event sanitizer -> subscriber and exporter dispatch

Tool execution callbacks and each execution-intercept next continuation return the canonical ToolExecutionResult { result, annotation }. A forwarding intercept must preserve both fields in ToolExecutionInterceptOutcome; Relay retains pending_marks separately. Tool sanitize-response guardrails receive only result. Scope-end event sanitizers govern the annotation after Relay projects it to category_profile.tool_result_annotation.