pwn-ai-agent-metrics

PWN::AI::Agent::Metrics

PWN::AI::Agent::Metrics is the telemetry layer of the pwn-ai learning loop. Every tool dispatch performed by PWN::AI::Agent::Loop is recorded here (name, success, duration, last error) and persisted to ~/.pwn/metrics.json. PromptBuilder re-injects a compact effectiveness summary into the system prompt on every turn, so the model gains awareness of which tools historically succeed vs. fail on THIS host and can adapt its tool selection accordingly. This is one half of the closed feedback loop that lets pwn-ai continuously make itself smarter (the other half is PWN::AI::Agent::Learning). PER-ENGINE SEGMENTATION ----------------------- A local Ollama model and a frontier model do NOT have the same per-tool success rate — blending them mis-advises the local model about itself. Every record now also increments an :engines[] sub-bucket; summary/to_context accept engine: to surface only that engine's telemetry so the TOOL EFFECTIVENESS block becomes a genuine per-engine learned policy.

When to use

Call PWN::AI::Agent::Metrics from pwn_eval when the task needs this module. Do not reimplement it in shell.

Methodologies

Generated from pwn/ai/agent/metrics.rb. Prefer the public class methods below. Class methods take (opts = {}) and read opts.

How to call

PWN::AI::Agent::Metrics.help
PWN::AI::Agent::Metrics.load(opts)

Public methods

• load
• save
• record
• append_jsonl
• summary
• snapshot
• to_context
• proxy_trust
• ucb
• thompson
• advantage
• prm_advantage
• prm_n
• record_step_reward
• record_judge
• judge_confidence
• judge_rate
• effective_rate
• changepoints
• record_calibration
• calibration
• calibration_green
• scale_prediction
• scoreboard
• health_line
• reset
• authors
• help
• calibration_green?

Source

pwn/ai/agent/metrics.rb

Verification

PWN::AI::Agent::Metrics.respond_to?(:load) after the module is loaded. Read the source for parameter names.