managed-model-endpoints

Managed model endpoints

A managed model endpoint is a model service the daemon owns: you register it once, then every compute_provider cell against it just works — the daemon swaps the resident model off the device (one model at a time, via the resident's own approved stop), runs your approved start script, waits for the readiness route, then runs your cell, streaming its lifecycle progress into the cell as it goes. You never run the container runtime yourself, never poll readiness in cells, and never see the credential value. Two verbs: register() (asks the user once) and ordinary inference cells.

Container specifics — image, registry login, internal port, cache mount target, readiness route — come from the model's own runbook skill; this skill is the translation contract.

Calling a registered endpoint — inference cells

Calling a registered endpoint — use the using-model-endpoint skill (this skill is the REGISTRATION contract; that one documents the call side in full).

The ONLY dispatch form is the compute_provider tool with the endpoint's registered name (list_compute shows them):

compute_provider(provider="boltz2-service", code="""
import requests
r = requests.post(BASE_URL + "/v1/infer", json=payload)
""")

The daemon brings the model up on demand (a first cold start downloads image + weights — minutes; let it run) and preloads BASE_URL into the cell — both as a Python variable (use it directly, as above) and as os.environ["BASE_URL"] (plus INFER_API_KEY for remote endpoints). Endpoints are not kernel environments: environment="boltz2-service" on a plain python cell fails — plain cells get no BASE_URL.

Enablement — once per machine

The user connects the family under Customize → Compute → Model endpoints (the setup flow saves the family credential first — connect-without-key is not a state) and picks ONE mode: Local (container registrations) or URL/remote (https against the configured host). Until connected, free_port()/register() raise a precise error — relay it; in the wrong mode they refuse with a teaching error naming the setting (existing endpoints of the unarmed leg keep dispatching — only NEW registrations refuse). Disconnecting is a full teardown: every active local service is stopped via its approved stop script and every registration (local AND hosted) is removed; caches stay on disk; a failing stop keeps that one row, FAILED. The "Local machine GPU" toggle never gates registration — it governs cell GPU access only; the approval card is the per-registration gate.