cubepi-trace

Debugging cubepi runs with cubepi trace

cubepi can record every agent run as an OpenTelemetry span tree, written to local JSONL files. The cubepi trace CLI reads those files so you can see exactly what happened inside a run without re-running it: which LLM calls and tool calls fired, in what order, what each returned, where it errored, and the token/cache counts. Reach for this before guessing at a bug.

When this applies

• The final answer is missing, truncated, or the run "ended" with no reply.
• A tool (web_search, web_fetch, execute, MCP tool, …) returned something wrong or empty, or the agent looped.
• The model returned a 4xx/5xx (e.g. BadRequestError), or the run status is error.
• Token / cache / cost numbers look wrong.
• You just need to understand the agent's actual trajectory for a given input.
• A background LLM call (e.g. memory consolidation via Tracer.oneshot()) isn't behaving as expected — oneshot calls produce invoke_agent spans too and are searchable by metadata.

Prerequisites (one-time)

Tracing must be on and recording content for the run you want to inspect. In cubebox it's config-driven (dynaconf):

• tracing.enabled: true
• tracing.directory: ./cubepi-traces (default; relative to the backend cwd)
• tracing.record_content: true (needed to see prompts / tool args / results; off = you get the span tree and timings but no content)

Env override form: CUBEBOX_TRACING__ENABLED=true, CUBEBOX_TRACING__RECORD_CONTENT=true, CUBEBOX_TRACING__DIRECTORY=.... cubebox's config.development already enables this. Files land at <directory>/<YYYY-MM-DD>/<trace_id>.jsonl — sharded by trace_id, so one file holds the whole trace, including any nested subagent runs (they inherit the parent's trace). A trace that crosses UTC midnight is split across two date dirs; the CLI merges them. (cubepi.run_id is still recorded as a per-span attribute if you need to distinguish individual runs within a trace.)