Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add DataSQRL/sqrl --skill handle-large-data-filesgit clone --depth 1 https://github.com/DataSQRL/sqrlWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/datasqrl/sqrl/handle-large-data-files)<a href="https://agentmods.dev/skills/datasqrl/sqrl/handle-large-data-files"><img src="https://agentmods.dev/badge/skills/datasqrl/sqrl/handle-large-data-files/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/datasqrl/sqrl/handle-large-data-files"><img src="https://agentmods.dev/badge/skills/datasqrl/sqrl/handle-large-data-files.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00048 | $0.00655 |
| Opus 5.5 | $0.00019 | $0.00262 |
| Sonnet 5.5 | $0.00010 | $0.00131 |
| Haiku 4.5 | $0.00005 | $0.00065 |
Grade A, and why
handle-large-data-files scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 34 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Handle large data files
DataSQRL copies every recognized data file (.csv, .csv.gz, .json, .jsonl, .parquet, .orc, .avro, ...) found anywhere in the project into Flink's build/ data directory on every compile and test — with no cache. A large file (tens of MB or more) therefore makes every run slow and can exceed the test timeout. compile/test is blocked while such a file is present (LARGE_DATA_BLOCKED).
Renaming a file to *.bak removes it from the copy: the compiler recognizes files by data extension, and .bak is not one, so a .bak file is skipped — and it no longer triggers the block.
For each large file, first invoke the /data-observation skill and observe it (so you understand its schema and values), then classify and act.
Steps to follow
(a) The file is data a connector reads
Replace it with a small, coherent sample and point the connector at the sample.
- Produce a small sample (<1000 rows), keeping the same format so the connector's
'format'still matches:- For
.jsonl/.csvfiles, usehead -1000 big.jsonl > sample.jsonlcommand - For
.csv.gzfiles, usezcat big.csv.gz | head -1000 | gzip > sample.csv.gzcommand - For
.parquetfiles, useduckdb -c "COPY (SELECT * FROM 'big.parquet' LIMIT 1000) TO 'sample.parquet' (FORMAT PARQUET)"command
- For
- Sample coherently across related files. When files reference each other (foreign keys), sample a consistent set of keys so references still resolve — otherwise the sampled test data has orphaned references. Pick a set of parent keys first, then keep only the child rows for those keys.
- Point the connector's
pathat the sample file. If you sampled into a different format than the original (e.g. parquet → jsonl), also update the connector's'format'(and any format-specific options) to match — the'format'key is independent of the file extension. - Rename the original large file to
*.bakso it is not copied, and delete any copy already written underbuild/.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 34 lines · 48 tokens per session scan A 794189373853
handle-large-data-files is a skill published in the GitHub repository DataSQRL/sqrl (230 stars, last pushed today), licensed Apache-2.0. It adds 48 tokens to every session and 655 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-30.
Other skills, from other repositories
grpc-expert
Expert-level gRPC, Protocol Buffers, microservices communication, and streaming. Use when the user mentions Protocol Buffers, microservices, RPC, or streaming, or when the task involves gRPC Fundamentals, Communication Patterns, or Production Features.
kafka-expert
Expert-level Apache Kafka, event streaming, Kafka Streams, and distributed messaging. Use when the user mentions streaming, messaging, event driven, or Kafka Streams.
data-ingestion-pipeline
Build data ingestion pipelines for batch and streaming data from multiple sources. Covers extraction strategies, format normalization, deduplication, validation gates, and staging patterns. Triggers on data ingestion, ETL pipeline, or data import architecture requests.
openrouter
OpenRouter unified AI API - Access 200+ LLMs through single interface with intelligent routing, streaming, cost optimization, and model fallbacks.
llm-integration
Claude and OpenAI API patterns, prompt design, context management, streaming, tool use, cost control, multi-model routing.
fastapi-patterns
FastAPI production patterns — routing, dependency injection, background tasks, streaming, error handling, and async. Use when building or reviewing a FastAPI service.