ContextJet-ai

ContextJet-ai/awesome-llm-observability

Plugin Claude Code

Plugin marketplace listing 1 plugin: llm-observability.

ContextJet-ai/awesome-llm-observability

Plugin Claude Code

26 Agent Skills (several with runnable, unit-tested scripts) for building, evaluating, securing, and monitoring reliable LLM & AI-agent apps.

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI)…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages. Trigger on "handle LLM API errors", "add retries/fallbacks", "the app breaks when OpenAI is down", "rate limit errors", "make my LLM calls reliable", "timeout handling". Add retries, timeouts, and model/provider fallbacks…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to add safety and security guardrails to an LLM/agent app - blocking prompt injection, PII leakage, jailbreaks, toxic output, off-topic responses, or invalid structured output. Trigger on "add guardrails", "prevent prompt injection", "stop PII leaks", "validate the model's output", "make this safe for…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy golden dataset. Trigger on "review my LLM outputs", "have an expert label these", "error analysis", "annotate traces", "build a golden dataset", or when automated…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to build a good evaluation dataset for an LLM app, the part everyone underestimates. Trigger on "make an eval set", "what should I test my LLM on", "I don't have test data for my prompt", "build a golden dataset", or before setting up evals. A great eval set beats a great metric; garbage-in means your evals…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-consistency check".…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to recommend an LLM observability / evaluation tool or stack for a specific situation. Trigger on "which observability tool should I use", "compare Langfuse vs Phoenix vs LangSmith", "what's the best LLM monitoring for us", or picking an eval/tracing/gateway tool given constraints (self-hosting, budget…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release. Trigger on "which model should I use", "is GPT/Claude/Gemini/Llama better for this", "should I switch models", "can a cheaper model do this", "compare models for my use case". Evaluate on YOUR task, not on…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to diagnose WHY an LLM agent or chain produced a wrong, empty, slow, or expensive result, by reading its observability trace. Trigger on "my agent gave the wrong answer", "the chain returned nothing", "why is this so slow/expensive", "debug this trace/run", or when a trace tree is available. Walk the span…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to detect when an LLM is making things up, so you can flag or block confident-but-wrong answers before users see them. Trigger on "detect hallucinations", "is the model making this up", "flag unreliable answers", "hallucination check", "confidence scoring for LLM output", or hardening a RAG/QA system. Pick a…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to estimate what an LLM call or feature will cost, and to compare models on price, before or after shipping. Trigger on "how much will this cost", "estimate my OpenAI/Anthropic bill", "is a cheaper model worth it", "cost of this prompt", "project my LLM spend". Ships a runnable, tested calculator so the…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any prompt/model/RAG…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to measure whether an AI agent actually completed its task end to end, not just whether individual LLM calls looked fine. Trigger on "is my agent working", "measure agent success rate", "evaluate my agent", "how good is my agent", "agent completion rate", or evaluating a multi-step/tool-using agent. Score the…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG", "improve RAG accuracy". Diagnose whether…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to improve a prompt systematically instead of hand-tweaking it by feel. Trigger on "optimize my prompt", "make this prompt better", "the prompt isn't working well", "auto-tune my prompt", "few-shot example selection", or when prompt quality has plateaued. Optimize against an eval set with a method, and let…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to adversarially test an LLM/agent app before attackers do - prompt injection, jailbreaks, data exfiltration, tool misuse, and unsafe output. Trigger on "red team my LLM", "test for prompt injection", "is my agent secure", "jailbreak testing", "security review of my AI app", especially before shipping…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this when adding LLM observability to an app that handles sensitive data (finance, healthcare, PII) and you must NOT ship raw prompts/PII to a third-party tracing backend. Trigger on "redact traces", "PII in observability", "can we self-host tracing for compliance", "GDPR/HIPAA/SOC2 and LLM logging", or…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to cut the cost of an LLM app using observability data. Trigger on "my OpenAI/Anthropic bill is too high", "reduce token usage", "the app is expensive", "optimize LLM cost", "why am I spending so much on the API". Find the expensive spans first (measure), then apply the cheapest wins. Don't guess - the trace…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to strip PII (emails, credit cards, SSNs, phone numbers, IPs, IBANs) out of text before it is logged to a tracing backend or sent to a third-party model. Trigger on "redact PII", "scrub sensitive data", "mask PII before logging", "don't send customer data to the tracing tool", especially for…

ContextJet-ai/awesome-llm-observability

Skill Claude CodeCodex

Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Measure impact on real…