cloud-inference-routing-workflow

Cloud Inference Routing Workflow

Purpose

Choose the smallest correct cloud AI inference path before changing provider resources, spending GPU budget, moving model weights, or deploying endpoints.

The practical decision is whether the agent should use an official provider plugin, provider MCP server, provider CLI, SDK, managed inference endpoint, rented GPU instance, cluster platform, or a stack-owned implementation workflow.

Source Check

Use repo-local model code, Dockerfiles, provider configs, checked-out scripts, installed provider plugins, installed provider CLIs, bundled MCP servers, and official provider documentation before making claims about current behavior.

Preferred official sources:

• Runpod docs and bundled MCP servers for Runpod Pods, Serverless, Flash, templates, volumes, registries, and docs lookup.
• Hugging Face plugin and huggingface_hub CLI for Hub repos, models, datasets, Spaces, Inference Endpoints, jobs, and papers.
• AWS Agent Toolkit, AWS CLI, AWS SDKs, and AWS docs for Lambda, SageMaker, Bedrock, ECS, EKS, Batch, IAM, billing, and observability.
• Vast.ai docs and CLI for low-cost rented GPU instances.
• CoreWeave docs for Kubernetes-backed GPU infrastructure, clusters, storage, networking, and observability.

Translate documentation into the concrete project decision it changes.

Routing Workflow

1. Inspect the workload:
   • task: inference endpoint, batch inference, fine-tuning, full training, eval, model conversion, quantization, benchmark, notebook, or one-off GPU shell
   • model family, parameter count, precision, context length, adapters, tokenizer, and required runtime
   • expected VRAM, CPU RAM, disk, network, container image, driver, CUDA, and startup time
   • latency, throughput, concurrency, cold-start tolerance, uptime, privacy, and cleanup needs
   • artifact flow: model weights, private datasets, outputs, logs, checkpoints, and caches
2. Choose the provider lane:
   • quick managed inference
   • quick custom endpoint
   • cheap flexible GPU box
   • training or conversion workspace
   • Kubernetes or production cluster
   • existing provider familiarity
3. Prefer familiar first-party paths when they fit:
   • Runpod for fast GPU Pods, Serverless endpoints, Flash, cheap/flexible experiments, templates, and resource management through MCP.
   • Hugging Face for model and dataset repos, conversion/publishing workflows, Inference Endpoints, Spaces, jobs, and Hub-native collaboration.
   • AWS when the project already uses AWS accounts, IAM, S3, CloudWatch, Lambda, SageMaker, Bedrock, ECS, EKS, or Batch.
4. Route to official tools:
   • Use runpod MCP for Runpod resource mutations and runpod-docs MCP for docs lookup when available.
   • Use the upstream Runpod skills under the exported skills/ tree for Runpod flash, runpodctl, and companion CLI details. Refresh those skills with npx skills update or npx skills add runpod/skills instead of hand-editing them.
   • Use the Hugging Face Codex plugin and Hugging Face CLI for Hub, model, dataset, Space, endpoint, or job work when available.
   • Use the AWS Agent Toolkit and AWS CLI or SDKs for AWS inference work when available.
   • Use provider CLIs or docs for Vast.ai and CoreWeave until a provider-owned agent surface is available or a concrete repeated Socket workflow justifies a new slice.
5. Confirm boundaries before mutation:
   • provider account, project, region, org, namespace, and budget
   • API key or profile source
   • model and dataset license or private-use boundary
   • exact resources to create, update, stop, delete, or scale
   • expected hourly cost, storage cost, bandwidth cost, idle behavior, and teardown path
6. Choose validation:
   • read-only docs, list, or pricing query for planning
   • local smoke test or CPU fallback before GPU spend
   • minimal GPU probe before full training or endpoint deployment
   • endpoint health, latency, throughput, logs, and cost check after deployment
   • cleanup verification for stopped Pods, terminated instances, deleted endpoints, detached volumes, and lingering storage