workerllm-runtime

workerllm-runtime

When you're authoring a produced app that needs to call an LLM at runtime — not the John-equipped build session itself, but the app's own runtime when its end-users use it — wire it to John's local LLM client server. The client is OpenAI-compatible; the same SDK that points at api.openai.com works against $JOHN_LLM_CLIENT_URL.

When to use this (vs alternatives)

• Use this skill for standalone produced apps that need workerLLMs at runtime. Examples: a doc-verification rule's check_R<id>.py that asks a model for a judgment call; a slide-renderer that asks for a one-sentence summary; a chatbot's main loop.
• Hosted-platform deployments are template territory. If the produced app is destined to run inside a hosted multi-tenant platform (proxy-mediated keys, metered billing), the platform's template supplies that pattern. Because this skill's call shape is plain OpenAI-compatible, migrating is just changing base_url — design the app so that's the only thing that moves.
• Don't use this for build-session subagent dispatch. That's [[subagent-dispatch]] — use the coding runtime's agent mechanism, not LLM APIs.

The call shape

The produced app uses the standard openai Python SDK pointed at the local client:

import os
from openai import OpenAI

client = OpenAI(
    api_key="not-used",  # the local client doesn't check; the workspace .env has the real keys
    base_url=os.environ.get("JOHN_LLM_CLIENT_URL", "http://localhost:8500") + "/v1",
)

resp = client.chat.completions.create(
    model="deepseek-v4-flash",  # see "Model selection" below
    messages=[
        {"role": "system", "content": "You verify loan-advertising compliance against Chinese regulation R012."},
        {"role": "user", "content": ad_text},
    ],
    response_format={"type": "json_object"},  # if you want structured output
    temperature=0.1,  # low for verification; raise for creative tasks
    max_tokens=1000,
)
verdict = resp.choices[0].message.content