Reuse
Borrowing it
Nothing to install: this file belongs to yaalalabs/agent-kernel. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/yaalalabs/agent-kernel/develop/.agents/skills/ak-dev-testing-conventions/SKILL.mdgit clone --depth 1 https://github.com/yaalalabs/agent-kernelWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/yaalalabs/agent-kernel/ak-dev-testing-conventions)<a href="https://agentmods.dev/skills/yaalalabs/agent-kernel/ak-dev-testing-conventions"><img src="https://agentmods.dev/badge/skills/yaalalabs/agent-kernel/ak-dev-testing-conventions/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/yaalalabs/agent-kernel/ak-dev-testing-conventions"><img src="https://agentmods.dev/badge/skills/yaalalabs/agent-kernel/ak-dev-testing-conventions.svg" alt="Reviewed on agentmods" width="80" height="20"></a> Per session 59 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 17,864 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
- NVIDIA SkillSpector pass
Origin original No closer match found in the catalogue.
Token cost
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00059 | $0.17864 |
| Opus 5.5 | $0.00024 | $0.07146 |
| Sonnet 5.5 | $0.00012 | $0.03573 |
| Haiku 4.5 | $0.00006 | $0.01786 |
Security
Grade A, and why
ak-dev-testing-conventions scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured today.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
.agents/skills/ak-dev-testing-conventions/SKILL.md · 529 lines
How it starts
The opening of the file, as written. The whole thing — 529 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Testing Conventions
Running Tests
cd ak-py
uv run pytest # Run all tests with coverage
uv run pytest tests/test_runtime.py # Run specific test file
uv run pytest -k "test_session" # Run tests matching pattern
uv run pytest -x # Stop on first failure
Coverage and HTML reports are auto-generated per pyproject.toml:
[tool.pytest.ini_options]
addopts = "--cov=src --cov-report=term --cov-report=html --html=report.html"
Test File Organization
Tests live in ak-py/tests/ and follow the naming convention test_<module>.py.
Four subsystems ship a reusable contract suite a new backend must be run against, rather than
per-backend assertions written from scratch: QueueTransportContract (pipeline/testing.py),
SandboxProviderContract (sandbox/testing.py), KnowledgeBaseContract /
DocumentStoreContract (knowledgebase/testing.py), and TraceFetchContract (trace/testing.py,
for tracers that implement BaseTrace.fetch_spans). All four ship next to the ABC they constrain,
so a bring-your-own backend author can import and subclass them out of tree; each module imports
pytest and is therefore kept out of its package's exports. In all four cases the contract classes
are deliberately not named Test*, so pytest collects them only through the subclasses that supply a
fixture.
| Test File | Tests |
|---|---|
test_base.py |
Session, Agent, Runner abstractions; Agent.run_options / RESERVED_RUN_OPTIONS / validate_run_options and Runner._native_kwargs (#754); the RunOptionsFactory export, Agent.run_options_factory and resolve_run_options (no-factory copy, sync and async merge, argument identity, non-mapping TypeError, reserved-key ValueError, propagation, static dict untouched) (#758) |
test_runtime.py |
Runtime registration, execution, hooks |
test_stream_events.py |
core/event.py's StreamEvent discriminated union: every member round-trips through JSON (type discriminator), rejects an unknown type, and stays JSON/pickle-safe — scalars for every member except DataMessage, whose content is a JSONPayload dict/list |
test_payload.py |
PayloadCodec (core/util/payload.py): normalise coercing datetime/Decimal/UUID/set/bytes/non-string keys and raising on an unserialisable object, a NaN or a scalar top level; encode staying byte-identical to json.dumps for an already-safe body while converting what it would refuse; the convergence of from_output's two branches; and the two bypasses (model_copy replacing content, in-place mutation) pinned as tested behaviour |
test_a2ui_hook.py |
The a2ui capability: what is labelled (a text reply whose body parses as a JSON object or array) and what is skipped (prose, an AgentReplyAny, an image, a scalar body), the agents filter, and A2UIPostHookFactory resolving to the no-op when disabled, absent or failing — plus the streaming-mode warning. By name (#762): scopes (the one scoping rule, shared with _applies_to), A2UIPostHookFactory.labels (enabled and in scope, ignoring message_type: delta, False and logged on a config failure), the v0.9.1 EXTENSION_URI, and a2a_data (an A2UI object becomes a one-item list; a list or another media type as is) |
test_cross_surface_parity.py |
One labelled reply asserted to arrive with the same content and label on REST, WebSocket/async, the queue gate, streaming, AG-UI, MCP and thread history. The fixture carries a datetime and a Decimal, since parity on an already-safe payload proves nothing. MCP and A2A run over the ResponseBuilder-built body the queue carries (#780); A2A carries a data part labelled in media_type and metadata["mimeType"], its one allowed difference being the A2UI binding's one-item list for an object (#762) |
test_runtime_stream_events.py |
Runtime.stream()'s streaming contract (specs docs/specs/523-ag-ui-support/ and docs/specs/670-streaming-post-hooks/): a bare str from an unmigrated runner fails loudly as a pydantic ValidationError, every event reaches PostHook.on_stream_event(), a returned list emits N chunks and ends the chain, a single return of a different type raises TypeError, a hook returning None drops the whole chunk, delta is populated only for TextDelta and taken from the emitted event, StreamHalt closes open boundaries then yields one error chunk and stores no session, any other exception propagates, and the final chunk is a bare StreamChunk(done=True) |
test_hitl_models.py |
#606 vocabulary: ResumeDecision/ResumeSpec/AgentResumeRequestAny/AgentPausedReplyAny round trips, the derived content, payload accepting any JSON value but refusing a framework-native object, and RunPaused in the StreamEvent union |
test_paused_run_state.py |
PausedRunState over ak.paused_runs: add/get/list/clear, resolution by interruption id, duplicate ids within and across records, picklability of the opaque payload, a corrupted entry dropped rather than failing the rest, the one-time in-memory-store warning, and the per-runner scoping — clear_for_runner leaves another framework's pause alone, and runner is derived from Agent.current() when an adapter omits it |
test_runtime_resume.py |
Runtime dispatch to resume(), the six failure modes (all raised before the adapter's try, which would otherwise flatten them) including an agent rebound to another framework, that a failed resume keeps its record so the decision can be retried while a successful one is cleared by the adapter, and that the agent Runtime activated is the agent the runner receives |
test_runtime_resume_warnings.py |
The three diagnostics: a pre-hook dropping a decision, a hook halting a resume, a post-hook swallowing a RunPaused |
test_hitl_stream.py |
The paused stream's terminal sequence — RunPaused, boundary drain, done=True, session stored (unlike StreamHalt) — and that the runner's generator is closed on the early exit |
test_hitl_chat_service.py |
The chat envelope: prompt-or-resume, both request builders, the 202 + status: "PAUSED" body, and schedule + resume rejected ahead of _maybe_schedule |
test_hitl_guardrail.py |
Input guardrails read a human's decision text, since it reaches the model exactly as a prompt does |
test_agui_resume.py |
The AG-UI interrupt terminal outcome, resolution by interruption id with no run id, two runs outstanding, and the status mapping that does not flatten cancelled into denied |
test_stream_boundaries.py |
StreamBoundaryTracker (core/runtime.py, not a module of its own) directly: open/close pairing per kind, innermost-first drain order, drain() clearing, and the two malformed-sequence cases it tolerates (closing an id never opened, opening one twice) |
test_module.py |
Module load/unload, wrapping; Module.run_options merge, chaining, reserved-key and unloaded-agent errors, _native_agent_name override, the base-class pre_hook / post_hook resolving through _native_agent_name, and the guard that a subclass implementing only _wrap and load constructs (#754); the positional factory parameter (stored, replaced, beside keywords, agent= still accepted, a non-callable or a second positional factory raising TypeError and storing nothing, a keyword named factory treated as an option) (#758) |
test_session.py |
Session state, caches, context vars |
test_session_cache.py |
LRU SessionCache |
test_sessions_in_memory.py |
InMemorySessionStore |
test_sessions_redis.py |
RedisSessionStore missing-config error, shared RedisDriver retry exhaustion |
test_sessions_valkey.py |
ValkeySessionStore round trips (fake client), shared ValkeyDriver retry exhaustion |
test_sessions_dynamodb.py |
DynamoDBSessionStore Binary wrap/unwrap, missing-item skip (mocked driver) |
test_shared_drivers.py |
Shared DB drivers (core/util/driver/): retry scope, ping/reconnect, command surface, DynamoDB item-dict semantics, S3Driver object operations and its no-probe-on-connect contract |
test_multimodal_redis_store.py |
RedisAttachmentStore index TTL refresh, JSON round trip, pruning (mocked driver) |
test_multimodal_source_forms.py |
MultimodalPreHook attachment source-form classification (spec #523 §8): bare base64 and base64 data: URIs are described/stored/stripped; http(s):///s3:// and non-base64 data: URIs are retained undescribed; empty data: payloads are dropped |
test_thread_source_forms.py |
The same source forms through ConversationThreadManager.store_attachments (issue #669): base64 stored as bytes and replaced by an AgentRequestAttachmentRef; a remote reference recorded by url with empty data and its request passed through unreplaced; mixed/multiple attachments keeping order; plus an end-to-end guard that a URL image survives MultimodalPreHook to the agent |
test_multimodal_tools.py |
analyze_attachments content routing per stored-record shape: a remote record's url is sent as-is (an image as image_url, anything else named in text) and never wrapped as base64; a stored record's base64 becomes a data: URI, a PDF a file part, any other type a [Document: ...] line; plus the two early returns that never reach the store or the LLM |
test_config.py |
AKConfig loading, env vars; the execution.mode × message_type matrix, defaults and the four _ExecutionConfig predicates, and the removed mode: stream failing with STREAM_MIGRATION_MESSAGE from YAML and env (#641) |
test_test_config.py |
AKTestConfig (Test framework config) loading, defaults |
test_tool.py |
ToolContext, cache |
test_tool_openai.py |
OpenAI ToolBuilder |
test_tool_crewai.py |
CrewAI ToolBuilder |
test_tool_langgraph.py |
LangGraph ToolBuilder |
test_tool_adk.py |
Google ADK ToolBuilder; its runner-driving mock agents come from the file's _mock_agent() helper (#758) |
test_tool_smolagents.py |
Smolagents ToolBuilder |
test_tool_pydanticai.py |
Pydantic AI ToolBuilder |
test_tool_maf.py |
Microsoft Agent Framework ToolBuilder |
test_tool_claude.py |
Claude Agent SDK ToolBuilder: JSON schema from the signature, sync tools on a worker thread keeping the ToolContext, annotations |
test_openai_runner.py |
OpenAIRunner execution, error handling; run options forwarded to Runner.run / run_streamed, and a real RunHooks resolving Session.current() / Agent.current() through Runtime.run and Runtime.stream (#754); a factory-merged mapping wins at Runner.run / run_streamed, resolution is awaited once per method, a failing resolution becomes the error reply in run and propagates from stream, and a real factory declared through OpenAIModule.run_options runs per call (#758). Every mock agent handed to a runner comes from a _mock_agent() helper carrying run_options and an AsyncMock resolve_run_options returning it, the contract all runner test files share since #758 |
test_adk_runner.py |
GoogleADKRunner execution; reserved keys, RUNNER_CONSTRUCTOR_OPTIONS splitting constructor-bound options from run_async-bound ones, constructor options reaching the per-run Runner(...), run forwarding a caller RunConfig untouched, and stream copying a caller RunConfig to force streaming_mode=SSE with a once-per-runner warning (#754); _split_run_options and _setup_session_context taking the resolved mapping, run handing a factory-merged mapping to the setup (a factory plugins reaching the constructor) and forwarding its run_config, stream SSE-copying a factory run_config, resolution awaited once per method, and an empty request returning before resolving (#758) |
test_crewai_runner.py |
CrewAIRunner execution (mocked Crew kickoff); reserved keys, declared options reaching the per-run Crew(...) with verbose overridable, empty options reproducing today's kwargs, and the module resolving the agent by role and rejecting reserved keys (#754); a factory-merged max_rpm and verbose reaching Crew(...), resolution awaited once, and a rejected request returning before resolving (#758) |
test_smolagents_runner.py |
SmolagentsRunner execution, multimodal requests, error handling; reserved keys, max_steps forwarded beside reset, a bypassed reset/additional_args overwritten by the AK-owned values, additional_args dropped when there is no framework context, and the module resolving the agent by its native name (#754); a factory-merged max_steps reaching agent.run beside reset, resolution awaited once, and a rejected request returning before resolving (#758) |
test_pydanticai_runner.py |
PydanticAIRunner execution, structured output, BinarySerde session round-trip, multimodal wiring; reserved keys, declared options forwarded to run beside the AK-owned keys, stream dropping event_stream_handler with one warning while keeping the declared dict intact, and reserved keys rejected at declaration through the module (#754); a factory-merged usage_limits reaching run, the stream filter applied to the resolved mapping (_stream_run_options taking a mapping and leaving it untouched), resolution awaited once per method, and an empty request returning before resolving (#758) |
test_maf_runner.py |
MAFRunner execution, AgentSession snapshot round-trip, single user Message for text plus attachments, structured output, streaming event mapping incl. tool-call ids that arrive only on a call's first chunk; reserved keys, declared options forwarded beside the AK-owned keys; framework context round-trip (ak_context not stored twice); MAFAgent prompt/tool attachment; Human in the loop on a real MAF Agent with a fake chat client — pause without running the gated tool, approve/deny on a fresh agent from the pickled session, the refusals (partial, payload, prompt, new turn while paused), retry after a failed resume, streamed RunPaused/resume_stream, and dispatch through Runtime |
test_claude_runner.py |
ClaudeRunner with a fake ClaudeSDKClient replaying real SDK messages: transcript kept on the AK session through the SessionStore (first turn session_id, later resume, commit only after a result, BinarySerde round-trip), stream mapping, tool server and approvals, subagents, multimodal blocks; reserved and unknown run options |
test_langgraph_runner.py |
LangGraphRunner execution; _merge_run_config deep-merge semantics (thread_id wins inside configurable, list-valued keys like callbacks concatenate AK-first, a scalar AK value wins over a None caller config, neither input is mutated), run/stream forwarding the merged config and other options to ainvoke/astream_events, the declared options surviving unmutated across runs, reserved keys, the nested config.configurable.thread_id and top-level RunnableConfig keys rejected at declaration, and aget_state receiving the merged config (#754); a factory-merged config deep-merged in run and stream with aget_state reading the same merged config back, resolution awaited once per method, and a rejected request returning before resolving (#758) |
test_trace_langfuse_langgraph.py |
LangFuse LangGraph trace runner; a caller-supplied callbacks list is appended after the Langfuse handler rather than replacing it (#754); a factory-merged config reaching ainvoke through the traced runner with the Langfuse handler kept first (#758) |
test_langgraph_reasoning_live.py |
LangGraph reasoning against a REAL reasoning model, env-gated (AK_TEST_REASONING_MODEL; skipped in normal runs). Guards the premise the chunk-feeding unit tests cannot: that the model streams a summary at all (it must be asked — reasoning={"summary": "auto"}) and that LangChain surfaces it under content_blocks. Builds a bare StateGraph rather than create_react_agent, since the test needs only one model call and a ReAct loop would add nothing |
test_guardrail.py |
Guardrail factories, hooks |
test_api_http.py |
REST API handler; TestUnqueuedProtocolRefusal (#780): RESTAPI.agent_protocol_surfaces table, and RESTAPI.run refusing A2A/agent MCP with explicit handlers, on a broker and from a subclass, while MCP serving only app tools still serves and a plain in_memory run still hands off to IOHandler |
test_api_mcp.py |
MCP.get_http_app(): mcp.stateless_http is passed to http_app() (fastmcp 3 removed the FastMCP(stateless_http=...) constructor kwarg, so the constructor must stay name-only), and a second call reuses the built server while still applying the configured mode. The executor's queued replies (#780): a fake client set on MCP._client answers with a ResponseBuilder-built body (the bytes a runner stores), covering labelled object/array ToolResults, text and image text, the request sent (protocol="mcp"), and every failure (error status, missing result, timeout, exception) as a ToolError with a fixed message, end to end through a real fastmcp Client |
test_api_a2a.py |
The A2A executor through the queue (#780), on a2a-sdk 1.x (#762), with a fake client on A2A._client, a real 1.x RequestContext (ServerCallContext() first) and a RecordingEventQueue (1.x's EventQueue is abstract): one text message per unlabelled reply (text, image, structured as JSON text), ids passed by keyword (the text helper's second parameter is media_type), TestDataParts for a labelled reply (one data part labelled in media_type and metadata["mimeType"], an A2UI object as a one-item list, another media type's object kept, 1 arriving as the float 1.0), every failure as a raised InternalError carrying the generic message (or TIMEOUT_MESSAGE) with no event enqueued, cancel raising UnsupportedOperationError, the body sent, the context checks, get_client() built once, and importing the module never loading pipeline.request_handler |
test_api_a2a_surface.py |
The A2A surface on a2a-sdk 1.x (#762): agentkernel.api.a2a.a2a and .handler import (the regression no test caught); A2ACardBuilder (one 1.0 HTTP+JSON interface at {a2a.url}/{name}, no url/preferredTransport in the served card, text/plain modes, the A2UI media type and v0.9.1 extension only for agents A2UIPostHookFactory.labels, a non-string description emptied with a warning, untagged skills tagged with the agent name); and the routes over HTTP through RESTAPI.build_app — 1.0 routes answer, /v1/... and /{tenant}/... are FastAPI's 404, the well-known card is camelCase with an ETag, an unknown agent's card 404s, the catalog lists the served cards, and a card-less agent is skipped with a warning |
test_chat_service_core.py |
ChatService execution core (execute/execute_stream): typed replies, prebuilt request lists, validation, error propagation, wrapper wire shapes |
test_chat_service_streaming.py |
ChatService SSE/stream chunk formatting |
test_integration_adapter_contract.py |
IntegrationAdapterContract subclassed once per built-in messaging adapter: identifier resolution, ignorable deliveries, reply-context budget, queue round trip (start here for a new platform) |
test_slack_integration.py |
Slack adapter: event -> InboundRequest and reply -> Slack messages, plus Bolt's signed dispatch through the webhook host (pattern for per-platform adapter tests) |
test_integration_roundtrip.py |
A platform event through the whole in_memory topology to a recording outbound adapter |
test_whatsapp_integration.py |
WhatsApp adapter: webhook delivery -> InboundRequest, and agent reply -> Cloud API sends |
test_teams_integration.py |
Teams adapter: Bot Framework activity -> InboundRequest (through process_activity), and agent reply -> proactive continue_conversation, including the per-delivery BotFrameworkAdapter |
test_gmail_integration.py |
Gmail adapter: unread mail -> InboundRequest, and agent reply -> a threaded reply |
test_messenger_integration.py, test_instagram_integration.py |
Messenger / Instagram adapters: webhook delivery -> InboundRequest, and agent reply -> Send API calls |
test_telegram_integration.py |
Telegram adapter: update -> InboundRequest, and agent reply -> Bot API sends |
test_integration_adapter_factory.py |
IntegrationAdapterFactory: how an integration attribute value becomes an outbound adapter — bare names, per-platform dotted-path overrides, instance caching, and the configuration errors (unknown name, wrong type, unimportable path, missing extra); an inbound adapter is never resolved by name |
test_integration_producer.py |
IntegrationProducer: what a parsed platform message looks like on the input queue — routing attribute, reply_-prefixed context and its 8 KB budget, ordering/dedup keys from the platform ids, prebuilt request list in the body |
test_integration_webhook_handler.py |
WebhookRESTRequestHandler: the generic host for a push-based inbound adapter — route mounting, batched deliveries, edge acknowledgement, ignored deliveries, SDK-owned responses, verification rejection, and RESTAPI.run refusing it |
test_integration_poller_runner.py |
PollerRunner: hosting a pull-based inbound adapter — enqueue-then-mark_handled ordering, a failing poll costing one iteration, shutdown within one interval, and the topology rejections both ways |
test_attachment_offload.py |
AttachmentStorageManager.offload: image/file requests rewritten to refs in place, other requests kept in order, the caller's message when multimodal is disabled, session_cache refused for an attachment but not for a text-only list |
test_thread_integration.py |
Thread integration: ThreadRecorder ordering/enforcement, AgentThreadRequestHandler recording + no-phantom-thread prechecks, stream accumulation, end-to-end read-back. Plus the #763 shape: ThreadMessage keeping a JSON-looking string a string while normalising a dict exactly as AgentReplyAny.content does, refusing both half-states (object without a label, label over text), still reading a row written before media_type existed, and to_wire() omitting an unset label while keeping an attachment's name: null — the guard for naming the excluded field rather than using exclude_none. _as_thread_message is asserted per caller shape, and one end-to-end thread holds a labelled and an unlabelled reply so the unlabelled one pins what an application that never opted into A2UI still reads |
test_thread_pipeline_recording.py |
Thread recording split across the queue: ThreadRequestHandler marking the message and committing the user message (and offloading attachments) before enqueue, its rejections leaving no phantom thread (missing user_id, unavailable agent, async mode, rest_sync + delta without a store that can serve SSE), deferred requests staying unmarked, AgentRunner/StreamAgentRunner appending the reply only for a marked message and only after the output send, a thread-store failure never retrying the run, IOHandler.run(request_handler=...) replacing rather than joining the chat route and refusing a non-RequestHandler, and the end-to-end read-back. The two dict cases are deliberately separate (#763): a dict with no label is still recorded as JSON text rather than a Python repr, while a response body carrying media_type beside result records the object and the label — the second is what pins the queue half, since the label travels in a sibling key the runner used to drop |
test_thread_router.py |
Thread read routes (ThreadRESTRequestHandler): pagination, Authoriser 401/403 semantics |
test_authoriser_shared.py |
Shared Authoriser in agentkernel.auth: package-export identity, guard that the thread package no longer exposes it, AuthValidatorAuthoriser adapter, AuthorisedRESTRequestHandler inheritance |
test_akagentrunner_stream.py |
Serverless ServerlessStreamAgentRunner (SQS streaming) |
test_serverless_agent_runner_schedule.py |
Serverless runners' trigger consumption: request_id/user_id body fallback, attribute precedence, missing-in-both error path |
test_akresponsehandler.py |
Serverless response handler (CHAT_RESPONSE / STREAM_CHUNK broadcast) |
test_serverless_integration_delivery.py |
Messaging-integration traffic through the serverless consumers (#760, Lambda-shaped records): the runners pass the prebuilt requests and copy integration + reply_* onto the output (non-integration output attributes unchanged), the stream runner hands integration records to the non-streaming path, the response handler delivers in all four modes without touching the store or the WebSocket, error statuses, retries via batchItemFailures, permanent failures (the MessageGroupId system-attribute KeyError regression), a missing platform extra, import hygiene, the round trip over a real SQSTransport (at most 10 MessageAttributes), and the end-to-end signed Slack event through Lambda.handler |
test_lambda_event_translator.py |
LambdaEventTranslator: API Gateway v1 event → Starlette Request (byte-exact plain and base64 bodies, multi-value headers and query with the single-value fallback, a Meta handshake in the single-value shape, scope fields) and results/exceptions → proxy responses matching FastAPI (Response pass-through, jsonable_encoder values, base64 bodies, HTTPException status/headers/detail, generic 500) |
test_lambda_webhook_host.py |
LambdaWebhookHost / LambdaWebhookGuard: the host's handle/challenge registered with Lambda.register and reached through Lambda.handler (a rejected delivery keeps its 403), one reused event loop never set as the thread's current loop, every cold-start guard at host construction (transport, mode, base-path variables, per-adapter missing_verification_settings, non-webhook handlers), and the test_integration_webhook_handler.py cases replayed through the host |
test_lambda_webhook_parity.py |
Every built-in webhook adapter answers the same through LambdaWebhookHost as through the FastAPI route: status, body bytes and enqueued message, for valid, unauthentic, ignored and handshake deliveries (Meta/Telegram from the contract subclasses, Slack and Teams HTTP-level) |
test_authorizer_webhook_bypass.py |
WebhookRouteMatcher (method + path only, strips the base path exactly as the router does, fails closed without it) and APIGatewayAuthorizer(bypass=...) (exact-methodArn Allow with principal integration:<name>, validator never called, spoofed headers and undeclared methods still validated, a raising bypass falling through) |
test_ws_lambda_stream.py |
WebSocket Lambda router with async + delta |
test_cli_tester.py |
CLI test framework |
test_auth_handler.py |
Auth handler |
test_akauthorizer.py |
AWS Lambda authorizer |
test_lambda_router.py |
Lambda routing; the REST router refusing rest_sync + delta at init, with and without queues (#641) |
test_aws_lazy_exports.py |
Lazy exports and import isolation (the sys.modules pattern): agentkernel.aws resolves every export without loading the containerized package; touching Lambda loads neither FastAPI, Starlette nor the webhook host; the authorizer and WebhookRouteMatcher load neither FastAPI nor slack_bolt; importing WebhookRESTRequestHandler loads neither agentkernel.api.http nor uvicorn (#760) |
test_packaging_extras.py |
Static pyproject.toml guard: the six webhook platform extras ship fastapi, slack ships aiohttp, gmail stays fastapi-free (#760); the a2a extra pins exactly a2a-sdk[http-server]==1.2.3 and ships fastapi and uvicorn, which 1.x's http-server extra no longer brings (#762) |
test_sqs_handler.py |
AWS SQSHandler config, client, message sending |
test_serverless_request_handle.py |
BaseRequest/BaseRunRequest parsing from serverless payloads |
test_firestore_database_id.py |
Shared FirestoreDriver (core/util/driver/firestore.py, explicit constructor params) named database_id configuration |
test_ak_logger.py |
AKLogger level resolution, configuration |
test_error_util.py |
user_facing_error_message error mapping |
test_thread_runner.py |
ThreadRunner task validation, failure/shutdown semantics |
test_ecs_sqs_consumer_parallel.py |
ECSSQSConsumer message processing + delete/retry semantics |
test_ecs_agent_runner_schedule.py |
ECS runner trigger consumption: request_id/user_id body fallback with attribute precedence, and ChatService's status forwarded to the output queue instead of discarded |
test_ecs_output_consumer_status.py |
ECS output consumer persisting status_code on stored records (default 200, permanent failure 500) |
test_deployment_queue_contracts.py |
#495 public-interface cleanup: pipeline.transport (QueueTransport/QueueMessage) is the only public queue API; RawQueueConsumer + SQSHandler's send models are internal; removed public names (QueueHandler, QueueConsumer, deployment.common.queue_*) raise ImportError |
test_pipeline_agent_runner.py |
AgentRunner/StreamAgentRunner: chat execution via ChatService, reply forwarding with STATUS_CODE attribute, per-chunk dedup suffixes, run() rejecting in_memory transport |
test_pipeline_agent_runner_schedule.py |
Pipeline runners' trigger consumption: request_id/user_id resolved from the message body, attribute precedence, and body-resolved metadata injected back into the attributes for output forwarding |
test_pipeline_bookkeeping.py |
Delivery bookkeeping for transports lacking native receive counts/dedup (spec #495 §6): InMemoryBookkeepingStore/RedisLikeBookkeepingStore attempt counters, retry-safe dedup claims, BookkeepingStoreFactory backend selection |
test_pipeline_sqs_transport.py |
SQSTransport: send/fetch/ack/nack/dead_letter, graceful shutdown handling, fetch-wait slicing, shared wire-format primitives |
test_pipeline_kafka_transport.py |
KafkaTransport against a fake in-memory Kafka cluster: send/fetch/ack/nack, dead-letter routing scoped by topic, delivery errors, consumer capacity check |
test_pipeline_nats_transport.py |
NatsTransport against a fake JetStream behind a real _NatsLoop bridge: subject/header construction, stable client-side (crc32) partition hashing, num_delivered mapping, nak redelivery, term() on permanent failure, one-in-flight-per-partition, stream-scoped dedup, auto_provision create-vs-verify, consumer capacity warning, and the full QueueTransportContract with no skips |
test_response_store_in_memory.py |
InMemoryResponseStore: get_record (status_code exposed), add_chunk/stream chunk-streaming for local SSE |
test_transport_contract.py |
The reusable QueueTransportContract (pipeline/testing.py) run against the in_memory transport |
test_transport_contract_live.py |
The same contract against REAL brokers, env-gated (AK_TEST_NATS_URL / AK_TEST_KAFKA_BOOTSTRAP; skipped in normal runs) with per-test unique streams/topics; run in CI by test-reusable.yaml's transport-integration-tests job over the transport examples' compose stacks. Documents two live-only timing traps: the per-partition pull window must stay below ack_wait, and partition counts are chosen from the real partitioner mappings (crc32 / murmur2) |
test_pipeline_request_handler.py |
Pipeline RequestHandler over FastAPI TestClient: rest_sync parity (stored status_code honored), rest_async accept/poll, SSE streaming end-to-end, multipart-on-in_memory only, and SSE error frames carrying handler-owned text only — a store/driver TimeoutError/Exception message never reaches the client (CodeQL py/stack-trace-exposure); the wait budget (60 × 1 s without a block) and loop, with subclass overrides of the store and budget still honored; RestHandler.request (#780): the envelope ({request_id, protocol} attributes, session group, request-id dedup, exclude_none body), validation with nothing sent, the wait (first read before any sleep, retry_count reads and one sleep fewer, get_and_delete=True, no early return in rest_async), ReplyTimeoutError fields, the budget read from config, transport and store built once through the factories, and QueuedReply fields |
test_pipeline_protocol_routing.py |
The protocol attribute (#780): forwarded by AgentRunner; one whole reply (no chunks, no USER_ID needed on a broker) from StreamAgentRunner; the single-reply path, never the connection pool, from RealtimeAgentRunner; 500 bodies for tagged permanent failures; the ResponseHandler storing a tagged reply (and a tagged permanent failure as a 500 record) in every mode; integration winning over protocol; untagged WS replies still broadcast; QueueMessage.protocol_of (non-empty str only), and the runner side never loading pipeline.request_handler |
test_protocol_pipeline_roundtrip.py |
End to end over in_memory (#780): a pump thread drains both queues through AgentRunner/StreamAgentRunner and ResponseHandler; an A2A message:send (an a2a-sdk 1.x ClientFactory client over httpx.ASGITransport, with AK_A2A__URL pointed at the test's base URL since the factory calls the card's interface URL and sets the required A2A-Version: 1.0 header) and an MCP call_tool (fastmcp.Client(MCP.get())) run on the pump thread, keep their session across calls, and share one session across the two surfaces, in the default mode, async, and both delta pairs (rest_sync + delta, async + delta); a reply the A2UI hook labels on the runner arrives as an A2A data part in the default mode and rest_sync + delta (reset Runtime._system_post_hooks, which is cached per class), and the 0.3 /v1/message:send route is a 404 (#762) |
test_pipeline_response_handler.py |
Pipeline ResponseHandler delivery paths: REST records, the USER_ID-presence WS routing marker, STREAM_CHUNK/CHAT_RESPONSE pushes, missing-attribute retries, permanent-failure frames/records |
test_pipeline_integration_delivery.py |
IntegrationDelivery (#760): the return-address helpers (integration_of accepts only a non-empty str), reply-context stripping, result text vs the generic ERROR_MESSAGE for status >= 400, non-dict and missing bodies, failures propagating, and the lazy agentkernel.integration import |
test_pipeline_io_handler.py |
IOHandler.run() topology validation and fail-fasts (ASYNC-on-in_memory without a validator, broker WS modes without a push token, broker + non-shared response store, and — #780 — broker + A2A/agent MCP without a shared store in every mode), the no-agents warning (_warn_if_no_agents, startup carries on), signal handlers, graceful-drain exit code |
test_pipeline_ws.py |
The gateway tier: LocalConnectionRegistry, PodPushWebSocketHandler store-lookup delivery + stale-connection cleanup, /internal/push auth, the native /ws route (1008 closes, chat enqueue attributes, custom routes), WebSocketGateway fail-fasts, single-process async and async + delta end-to-end over in_memory, and cross-"pod" delivery between two gateway apps sharing one connection store |
test_session_connection_store.py |
The WSConnectionStore contract over the in-memory/redis-like/DynamoDB implementations, plus SessionStore.get_connection_store() per backend (store-less backends raise actionably) |
test_sandbox.py |
Sandbox core: model/capabilities (the package_ecosystems validator, derive()), error hierarchy, config (permission fields, package_installers, language_runtimes), provider contract, manager + factory + embedded broker, agent surface (system tools incl. the conditional install_packages + permission guidance, task-completion pre-hook), agents scoping |
test_sandbox_broker.py |
Broker flavors (embedded/thread) end-to-end, thread loop-identity contract, wait-policy promotion + late-completion recovery (including the bounded result_summary outcome on check_sandbox_task), completion ingestion |
test_sandbox_providers.py |
local_subprocess (real subprocess), docker (mocked SDK; language_runtimes drives the exec argv and instance languages, also on kubernetes and ec2_ssm), e2b/daytona (mocked cloud SDKs), ec2_ssm (mocked boto3), and kubernetes (fake SDK with a functional tar/base64 exec protocol; manifest shapes, NetworkPolicy gating, RBAC-impersonation headers per call shape) providers, run against the reusable SandboxProviderContract |
test_sandbox_queue_broker.py |
#503 queue broker: BrokerWireCodec binary round trips, error_type stamping, removed config fields, QueueExecutionBroker client (bounded waits, promotion, typed re-raise, ceiling/size guards, FIFO destroy), QueueBrokerWorker two-loop model (output-queue delivery, truncation, split permanent-failure hooks, sweep inventory), and manager-level wait-then-check_sandbox_task recovery over in_memory |
test_sandbox_permissions.py |
#692 permissions: SandboxPermissionGuard deny rules (one audit record, no code/package text), worker wiring (generic-warning skip on process()/ThreadBroker, capability-mismatch warning, legacy one-argument install_packages shim, effective timeout + backstop), ExecutionManager pre-submit checks (no side effects) and config warnings, the install_packages tool and its binding per framework adapter, installs end to end over embedded/thread/queue |
test_sandbox_packages.py |
#692 package specs: the PackageInstaller baseline, the python/node rules, PackageSpecRuleFactory resolution (built-in first, bring-your-own narrows), validate_config, argv()/command() shapes |
test_sandbox_egress_live.py |
#692 live egress gate: network_egress: allowlist against real e2b/daytona sandboxes (listed registry reachable, unlisted host blocked, install succeeds); skipped unless AK_TEST_SANDBOX_EGRESS=1 plus the provider's API key. A local fake runs the same contract every time |
test_pipeline_factory_seams.py |
#503 explicit-config seams: QueueTransportFactory/ResponseStoreFactory with an explicit block never read AKConfig (asserted loudly), default paths unchanged, response-store ttl override |
test_response_store_scan.py |
The optional ResponseStore key-scan capability (supports_key_scan/scan_records) across the four built-ins, ABC default opt-out |
test_authoriser_shared.py |
Shared Authoriser in agentkernel.auth: AuthValidatorAuthoriser adaptation, export identity, and the guard that the thread package no longer re-exports it |
test_schedule_model.py |
ScheduleSpec one-of/timezone/session_mode validation, chat-envelope parsing, ScheduledTask JSON round trip (JSON primitives only) |
test_schedule_manager.py |
ScheduleManager: get() gating + singleton, provider/transport fail-fast, semantic validation matrix (including the named-agent precheck and the unnamed-agent exemption), create ordering + rollback, trigger-body freezing, ownership, amendment rules (occurrence rule replaced as a unit, untouched when the amendment names none of it), cancellation, occurrence recording (never raises) |
test_schedule_store.py |
Every ScheduleStore backend against the one contract the in_memory class pins — in_memory, redis/valkey through a fake redis-like client injected as store._driver._client, dynamodb through a fake driver whose table.scan replays LastEvaluatedKey pages — plus index cleanup on delete, TTL behaviour (none by default), and ScheduleStoreBuilder built-in/BYO/unknown-type/missing-extra resolution |
test_schedule_provider_local.py |
LocalScheduleProvider: next-fire computation, one-time vs re-armed occurrences, token substitution, body-only delivery into InMemoryTransport, pause/delete disarm, ScheduleProviderFactory resolution |
test_schedule_provider_eventbridge.py |
EventBridgeScheduleProvider against a mocked boto3.client: cron/at expression translation, exact registration/amendment/removal kwargs sent to the Scheduler API, error mapping to ScheduleError, ScheduleProviderFactory wiring |
test_schedule_router.py |
ScheduleRESTRequestHandler: 404 when unconfigured, the three 401 variants from the shared AuthorisedRESTRequestHandler, listings forced to the authorised user, 403-before-404 ordering, PUT amendment happy path + validation 400s, DELETE returning the cancelled task, mount-time validation of the configured backends (get_router() building the manager, an unusable provider failing the mount, mounting while unconfigured still allowed), and the guard that importing the package does not pull in FastAPI |
test_secret_manager.py |
SecretManager: env → cache → provider precedence (empty env var is a miss, env hits never cached), SecretNotFoundError vs default, provider SecretError never masked, os.environ never written, key validation, TTL expiry with a fake clock, invalidate/clear, no lock held across the provider call, current() singleton built from config |
test_secret_factory.py |
SecretProviderFactory: env/kubernetes/aws_ssm/dotted-path resolution, unknown names and non-subclass paths rejected, missing aws extra raised before the prefix check, never reads AKConfig |
test_secret_providers.py |
EnvSecretProvider, KubernetesSecretProvider (tmp_path mounts: kubelet symlink layout, duplicates, depth, invalid UTF-8, missing mount) and AWSSMSecretProvider (mocked boto3: path composition, ParameterNotFound → None, other errors → SecretError, prefix validation), each run against the reusable SecretProviderContract (secret/testing.py) |
test_schedule_tools.py |
Schedule system tools: SystemToolFactory registration + schedule.agents scoping, prompt-suffix content, disabled short-circuit, acting-user read from the session volatile cache, and each tool's JSON contract including the no-identity and unknown-agent errors |
test_chat_service_schedule.py |
ChatService interception on all four entry points: 202 wire shapes, streaming terminal chunk, unconfigured 400, occurrence recording, no scheduling field leaking as AgentRequestAny |
test_serverless_status_propagation.py |
The status a chat run produced (e.g. 202 deferred-to-schedule, 4xx rejected) survives the serverless queue round trip: ServerlessAgentRunner forwards it as the ATTR_STATUS_CODE output-message attribute, ResponseHandler stores it on the response record, and the REST surface (DefaultEndpointsHandler, response-store polling) replays it to the caller |
test_pipeline_agent_runner_schedule.py |
Pipeline runners' trigger contract: request_id/user_id body fallback, attribute precedence, attribute injection for output forwarding |
test_ecs_agent_runner_schedule.py |
ECS runners' body fallback (_get_record_attributes with/without a parsed body) and status_code custom attribute |
test_ecs_output_consumer_status.py |
ECSOutputConsumer stores status_code (present / absent → 200 / permanent failure → 500) |
test_serverless_agent_runner_schedule.py |
Serverless runners' body fallback in both _get_record_attributes implementations |
test_knowledgebase_model.py |
KnowledgeCapabilities defaults and model_dump() shape, the record TypedDicts, and KnowledgeBase.validate_capabilities's two invariants (unreachable declaration, query ⇔ query_language) — asserting a bare capabilities object never validates on its own |
test_knowledgebase_base.py |
The reshaped ABC: three abstract members, the five capability-gated operations raising KnowledgeCapabilityError, the guard that the ABC exposes no read, schema() merge order with capabilities written last and unoverridable, _derived_schema(), and the fetch-gated [<id>] prefix in format_results() |
test_knowledgebase_builder.py |
KnowledgeBuilder gating: the tool matrix and emission order, search_kb's any-backend-declares-search gate, write_kb's always-emitted/per-call check and build(writable=False) withholding it, read_kb's routing between query() and search() (moved here off the ABC), capability mismatches returning an actionable string rather than raising, semantic_map resolution across queries/browse paths/fetch id segments, and the generic query/params write metadata |
test_knowledgebase_okf_roles.py |
The OKF role model: OKFRoleRegistry.from_config over literal dicts (single role, multi-database, a different role per database), OKFRole.writable as the one place the consumer-vs-writable distinction lives, may_write/writable_databases_for/has_any_role, and both AKConfigErrors — a database naming no agent, and one agent as both producer and curator of the same database — each naming the database |
test_knowledgebase_okf_config.py |
The okf block and OKFCapabilityManager: YAML and AK_OKF__* parsing, AKConfig.okf defaulting to None and a pre-change config file parsing unchanged, every field default, _resolve_store's three branches and both type/uri agreement failures, the read-only-store warning, and the laziness guarantees (validation walks no store; a backend walks one on first use) |
test_knowledgebase_okf_prompts.py |
OKFPromptComposer.compose from config alone with no backend constructed: the per-database line and its access marker, both mandates for an agent holding different roles in two databases, each mandate scoped under its own database, '' for an unnamed agent, and the navigation protocol present exactly once regardless of database count |
test_knowledgebase_okf_tools.py |
The OKF agent surface end to end over real bundles in tmp_path: the tool set per role, func.__name__ == SystemTool.name, read scoping falling out of the per-agent builder, the per-(agent, database) write refusal as a string, no tool carrying the prompt section (it arrives through SystemToolFactory.get_prompt_sections), SystemToolFactory wiring, and write permission keyed to the tool's owner across a handoff (a stale entry-agent ToolContext is ignored) |
test_knowledgebase_backends.py |
The three SDK-backed backends with their clients monkeypatched: the read→search/query renames, each declaration, Neo4j's generic-with-cypher_*-fallback write metadata, and Starburst's db_schema no longer shadowing schema() |
test_knowledgebase_stores.py |
DocumentStoreContract over a real tmp_path and a fake boto3 client, plus the containment matrix (.., absolute, normalising escapes, symlinks) and the global-lexicographic list() ordering case (a/z.md before ab/b.md) |
test_knowledgebase_okf_parser.py |
OKFParserUtil: frontmatter splitting, every diagnostic code reachable, trust derived from verified alone, staleness against an injected now, link extraction, the bounded field_tokens index, and a guard that the module pulls in no store or HTTP client |
test_knowledgebase_okf_manager.py |
OKFManager end to end: manifest walk and truncation, ranking determinism, browse-at-a-namespace with and without a curated index.md, write-through visibility, one walk under two concurrent boundary-crossing callers, and that nothing is ever filtered on trust or staleness |
test_knowledgebase_contract.py |
The reusable KnowledgeBaseContract (knowledgebase/testing.py) run against FakeKnowledgeBase in four capability shapes, OKFManager over a real local bundle, and the three SDK backends with their clients monkeypatched. The schema()-is-callable assertion is the Starburst-collision regression guard |
test_knowledgebase_okf_envelope.py |
The declared scale envelope: a generated 10,000-concept bundle keeps all 10,000, and tracemalloc allocations attributable to the manifest walk stay under a 25 KB per-concept budget — measured over a 2,000-concept slice, 250 MB projected at 10,000 (allocations, not RSS) |
test_knowledgebase_exports.py |
Every agentkernel.knowledgebase.__all__ name resolves through the PEP 562 lazy map, and chromadb/neo4j/trino/boto3 stay out of sys.modules on import — the gate a new export has to pass |
test_factory.py |
Shared pluggable-backend helpers (resolve_dotted, require_extra, AKConfigError) in core/util/factory.py |
test_store_builders.py |
Session/thread/multimodal store builders: fail-loud on unknown type, BYO dotted-path subclass resolution |
test_trace.py |
Trace factory built-in resolution, BYO dotted path, unknown-type error |
test_trace_cloudwatch.py |
CloudWatch trace provider: provider installed once or an existing SDK provider reused, X-Ray endpoint and SigV4 selection from AWS_REGION / OTEL_EXPORTER_OTLP_*, resource defaults, SigV4 request signing, the run span's session.id / input.value / output.value, session.id stamped onto instrumentor child spans, and the per-run flush on Lambda. Records spans with a real SDK TracerProvider + InMemorySpanExporter and patches trace.get_tracer_provider / set_tracer_provider, never installing a global provider |
test_trace_fetch.py |
Trace.fetch_spans / fetch_traces models and fetchers: FetchedTrace grouping (root, usage), SpanStatus mapping per provider, TraceQuery window defaults/validation, strict matches and empty-filter rejection, OTelSpanClassifier kind/IO/session/usage reads, keyword-vs-query arguments, the shared TraceFetcher paging (span identity is (trace_id, span_id), newest limit kept, cursor stop rules, time-window pages sized past the re-read boundary), and the Langfuse, Logfire, Traceloop and CloudWatch request translation and span mapping against fake clients |
test_trace_fetch_contract.py |
The reusable TraceFetchContract (trace/testing.py) run against an in-memory tracer and the Langfuse and Traceloop fetchers behind fake APIs |
test_trace_langfuse_flush.py |
Every Langfuse traced runner flushes after a successful and a failed run, only on Lambda, and a failing flush is logged rather than raised |
Changes
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- today Changed · +2 lines 7c32098112aa
- yesterday Changed · +23 lines b10a83d57b0e
- 2d ago Changed · +4 lines b0254c6b3bb9
- 3d ago Changed · +4 lines 6b180d0bdb67
- 8d ago Changed · +2 lines 5de33b4bde50
- 9d ago Changed · +2 lines 4b00c1478dc9
- 17d ago Changed c129593d9f88
- 23d ago Changed · +3 lines cb08ef222ea4
- 29d ago Changed · +17 lines e14f9d19161a
- 1mo ago First seen · 472 lines · 59 tokens per session scan A 49163a7f4ce1
ak-dev-testing-conventions is a skill published in the GitHub repository yaalalabs/agent-kernel (192 stars, last pushed yesterday), licensed Apache-2.0. It adds 59 tokens to every session and 17,864 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Related
ak-dev-testing-conventions compared with skills from other repositories
| Mod | Type | Stars | Last push | Licence | Scan | Tokens | Works with |
|---|---|---|---|---|---|---|---|
| ak-dev-testing-conventionsyaalalabs/agent-kernel | Skill | 192 | yesterday | Apache-2.0 | A | 59 | Claude Code, Codex |
| cloud-native-security-defensivej4flmao/agent-skills | Skill | 38 | 3d ago | MIT | A | 127 | Claude Code, Codex |
| notionArvo-AI/aurora | Skill | 454 | yesterday | Apache-2.0 | A | 23 | Claude Code |
| datadogArvo-AI/aurora | Skill | 454 | yesterday | Apache-2.0 | A | 28 | Claude Code |
| scalewayArvo-AI/aurora | Skill | 454 | yesterday | Apache-2.0 | A | 29 | Claude Code |
| terrasharkLukasNiessen/terrashark | Skill | 710 | 4d ago | MIT | A | 56 | Claude Code, Codex |
| isaac-automatorisaac-sim/IsaacAutomator | Skill | 270 | 11d ago | Apache-2.0 | A | 123 | Claude Code, Codex |