Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add vaquarkhan/data-engineering-agent-skills --skill spark-serverless-reliability-and-state-managementgit clone --depth 1 https://github.com/vaquarkhan/data-engineering-agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/vaquarkhan/data-engineering-agent-skills/spark-serverless-reliability-and-state-management)<a href="https://agentmods.dev/skills/vaquarkhan/data-engineering-agent-skills/spark-serverless-reliability-and-state-management"><img src="https://agentmods.dev/badge/skills/vaquarkhan/data-engineering-agent-skills/spark-serverless-reliability-and-state-management/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/vaquarkhan/data-engineering-agent-skills/spark-serverless-reliability-and-state-management"><img src="https://agentmods.dev/badge/skills/vaquarkhan/data-engineering-agent-skills/spark-serverless-reliability-and-state-management.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00069 | $0.00926 |
| Opus 5 | $0.00034 | $0.00463 |
| Sonnet 5 | $0.00014 | $0.00185 |
| Haiku 4.5 | $0.00007 | $0.00093 |
Grade A, and why
spark-serverless-reliability-and-state-management scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 91 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Spark Serverless Reliability And State Management
Overview
Serverless Spark hides cluster failures behind short-lived runtimes and opaque retries. Silent partial writes, orphaned checkpoints, and timeout-induced corruption are common when agents treat serverless Spark like a long-running cluster job. This skill forces explicit state boundaries, resumable progress, and cleanup before any publish path opens.
When to Use
- implementing or reviewing Spark on
AWS Lambda,AWS Glueserverless, or comparable short-lived runtimes - designing S3-backed checkpoints, progress markers, or staged write paths
- recovering from timeout, OOM, or mid-batch partial failure without double-counting
- translating IceGuard-style checkpoint and rollback patterns into agent workflows
- hardening batch Spark that must survive cold starts, memory limits, and hard execution ceilings
Do not use this for always-on EMR or Databricks clusters unless the job also runs in a serverless path with hard time limits.
Workflow
-
Classify execution risk before coding. Document:
- maximum runtime and memory ceiling
- input cardinality and partition fan-out
- whether output is append, merge, or overwrite
- downstream consumers that would see partial data
- whether the job is restartable from a known offset or partition set
-
Design a resumable checkpoint contract. Require:
- durable progress markers in object storage (for example
s3://.../checkpoints/{run_id}/) - explicit run identifiers tied to orchestration metadata
- idempotent write semantics at the target grain
- a manifest or
_SUCCESS-style gate before publish - separation between staging prefixes and publish prefixes
Load
references/spark-serverless-reliability-patterns.mdfor checkpoint layout and orphan-detection patterns. - durable progress markers in object storage (for example
-
Implement timeout-aware rollback. Before any publish:
- detect incomplete partitions or missing manifest files
- roll back staged outputs when the runtime approaches its ceiling
- persist last-good checkpoint state for resume
- block downstream publish when rollback is incomplete
- emit structured failure context (run id, partition range, bytes written)
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 91 lines · 69 tokens per session scan A 388cf08d6a71
spark-serverless-reliability-and-state-management is a skill published in the GitHub repository vaquarkhan/data-engineering-agent-skills (45 stars, last pushed 3mo ago), licensed MIT. It adds 69 tokens to every session and 926 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
ultralytics-platform
This skill should be used when user asks to "upload my model to Ultralytics Platform", "push this run to the platform", "upload a dataset to platform", "download a dataset from platform", "search platform datasets", "start cloud training", "train on platform GPUs", "export a model on platform", "deploy a model…
kafka-shadowtraffic
Generate a ShadowTraffic configuration to populate a Kafka topic with realistic synthetic data. Discovers the target topic, its key and value schemas, and the correct serializers from the live cluster via any attached Kafka MCP server, then writes a ready-to-run shadowtraffic-config.json and Docker command. Use when…
remote-gpu-trainer
Use when running, debugging, verifying, or delivering a deep-learning experiment on an owned or rented GPU, especially AutoDL or a remote SSH host; also use for Windows + Clash/Mihomo high-port SSH banner timeouts, fake-IP, or TUN routing interference. Covers launch, checkpoint/resume, detached monitoring…
infra-ragflow-ops
An operations guide for RAGFlow, an application that lets teams build systems that answer questions from a knowledge base. It covers service checks and the connected models, databases, vector stores, and file storage.
ray-data
Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.
agent-platform-deploy
Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check deployment status, verify serving endpoints, or clean up resources by undeploying models and deleting endpoints. Use when asked to deploy models on Agent Platform, list available Model Garden models, check if a model is…