query-loop-implementation

Query Loop Implementation

Core Idea

Build the loop as product infrastructure, not prompt glue:

ConversationManager -> QueryLoop -> ToolRuntime

• ConversationManager owns durable state: session id, messages, user settings, budget, persistence.
• QueryLoop owns one task turn: call model, detect tool calls, execute tools, append tool results, repeat.
• ToolRuntime owns registered tools: schemas, permission checks, execution, error formatting.

Use ReAct as the mental model:

Thought -> Action -> Observation -> Thought -> Answer

Implement it as structured API traffic:

model thinking/text -> tool_call -> tool_result -> next model call -> final text

Implementation Workflow

1. Inspect the user's stack and current LLM call site. Find where messages are built, where the model is called, and whether tool/function calling is already configured.

2. Introduce a minimal query loop. Keep the first version narrow: one model call function, one tool registry, explicit exit conditions.

3. Normalize message shapes. Use the provider's structured tool-call format when available. Avoid parsing free-form Action: text unless the provider has no function/tool-calling API.

4. Add tool execution safety. Validate tool input against a schema, apply permission checks for risky tools, wrap failures as tool results, and log every call.

5. Add exit and budget guards before expanding features. Always include maxTurns, timeout/cancel support, token/cost budget checks, and a fatal-error path.

6. Keep context-window strategy outside this skill. Accept messages as loop input and return updated messages, but leave trimming, retrieval, summarization, and compaction to a separate context-management layer.

Minimal Loop

Adapt this shape to the user's language and SDK:

async function runQueryLoop({
  initialMessages,
  model,
  tools,
  maxTurns = 10,
  signal,
}: {
  initialMessages: Message[]
  model: ModelClient
  tools: ToolRegistry
  maxTurns?: number
  signal?: AbortSignal
}) {
  let messages = [...initialMessages]

  for (let turn = 1; turn <= maxTurns; turn++) {
    if (signal?.aborted) return { status: "aborted", messages }

    const response = await model.generate({
      messages,
      tools: tools.definitions(),
      signal,
    })

    messages.push(response.message)

    const toolCalls = extractToolCalls(response.message)
    if (toolCalls.length === 0) {
      return {
        status: "completed",
        finalMessage: response.message,
        messages,
      }
    }

    for (const call of toolCalls) {
      const result = await tools.execute(call, { signal, messages })
      messages.push(makeToolResultMessage(call.id, result))
    }
  }

  return { status: "max_turns", messages }
}