debug-inference

Debug Inference

Diagnose why OpenShell inference is failing and recommend exact fix commands.

Use openshell CLI commands to inspect the active gateway, provider records, managed inference config, and sandbox behavior. Use a short sandbox probe when needed to confirm end-to-end routing.

Overview

OpenShell supports three inference paths. Diagnose the correct one first.

1. Managed inference through https://inference.local
   • Configured by openshell inference set
   • Shared by every sandbox on the active gateway
   • Credentials and model are injected by OpenShell
2. Direct external inference to hosts like api.openai.com
   • Controlled by network_policies
   • Requires the application to call the external host directly
   • Requires provider attachment and network access to be configured separately
3. System inference used by platform functions
   • Configured by openshell inference set --system
   • Uses the sandbox-system route
   • Consumed in-process by the sandbox supervisor and not exposed to sandbox user code through inference.local

For local or self-hosted engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments, the most common managed inference pattern is an openai provider with OPENAI_BASE_URL pointing at a host the gateway can reach.

Prerequisites

• openshell is on the PATH
• The active gateway is running
• You know the failing setup, or can infer it from commands and config

Tools Available

Use these commands first:

# Which gateway is active, and can the CLI reach it?
openshell status

# Show both the user-facing and system inference routes
openshell inference get

# Show only the supervisor-only system route
openshell inference get --system

# Inspect the provider record referenced by the relevant route
openshell provider get <provider-name>

# Inspect gateway topology details when remote/local confusion is suspected
openshell gateway info

# Run a minimal end-to-end probe from a sandbox
openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}'