hermes-performance

Hermes Performance (local model)

When Hermes runs a local model on a modest GPU, prefill (prompt processing) dominates latency. Every token of prompt and every tool schema in context is paid on every turn. So the levers are, in priority order: (1) run warm, (2) shrink what's in context, (3) keep the model resident, (4) pick the right serving backend. (This setup: apex-fast:latest on a Pascal P40 via Ollama at http://<OLLAMA_HOST>:11434/v1 — substitute your Ollama host.)

The levers (priority order)

1. Run warm, not cold (biggest, free)

The cold one-shot CLI (hermes chat -q) re-initializes MCP and reloads the model on every run — the slowest path. The warm gateway / interactive session is the daily driver. Benchmark and run real work on the warm path. A "slow Hermes" complaint is most often someone timing cold one-shots.

2. Trim toolsets per profile (shrinks prefill)

Every enabled toolset's schemas sit in the prompt and are re-prefilled each turn. Cut them to what the profile actually needs:

• hermes chat -t web (or per-call -t) to enable only what's needed,
• platform_toolsets / disabled_toolsets in the profile config to lock the set. Fewer schemas = less prefill = faster every turn. This is the highest-leverage context cut on a slow GPU.

3. Pin frequently-used tools past tool_search (the deferral tradeoff)

tool_search dynamic loading is ON: MCP schemas are deferred behind 3 bridge stubs and fetched on demand. It saves prefill tokens but adds a discovery round-trip. On a slow GPU the extra round-trip is often net-negative for tools the profile uses constantly. Pin those tools so they load up front instead of incurring a discovery turn. This is a genuine tradeoff — see references/tool-search-tradeoff.md before flipping it, and measure per workload.

4. Keep the model resident

Set OLLAMA_KEEP_ALIVE=-1 so the model stays loaded in VRAM and you don't pay a reload (a large, GPU-specific cost) on the first turn after idle.