What is datachain-knowledge?

A guide for maintaining a DataChain-based knowledge base about datasets, storage, and data pipelines. DataChain is a tool for organizing and processing data and its metadata.

Listing, exploring, querying, saving, or processing data in local storage and cloud buckets such as S3, Google Cloud Storage, and Azure Blob Storage.

It keeps data-related notes and intermediate files organized while applying consistent rules for exploring and processing stored data.

datachain-knowledge — Skill for Claude Code and Codex by datachain-ai

About the project

DataChain is a Python library and data-processing system that turns files from cloud storage or local filesystems into typed, versioned datasets that can be queried and processed. Teams use it to build repeatable pipelines for unstructured data, with optional knowledge-base and agent features that help coding agents understand and work with those datasets. The catalogue entries connect agents to DataChain's data workflows.

datachain-ai/datachain · 2,826 stars · on GitHub · docs.datachain.ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add datachain-ai/datachain --skill knowledge

Clone the repo

git clone --depth 1 https://github.com/datachain-ai/datachain

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/datachain-ai/datachain/knowledge/github.svg)](https://agentmods.dev/skills/datachain-ai/datachain/knowledge)

Your own site

<a href="https://agentmods.dev/skills/datachain-ai/datachain/knowledge"><img src="https://agentmods.dev/badge/skills/datachain-ai/datachain/knowledge/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/datachain-ai/datachain/knowledge"><img src="https://agentmods.dev/badge/skills/datachain-ai/datachain/knowledge.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 104 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 3,140 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits

NVIDIA SkillSpector pass 7 Sept 2026

How audits are shown

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00104	$0.03140
Opus 5.5	$0.00042	$0.01256
Sonnet 5.5	$0.00021	$0.00628
Haiku 4.5	$0.00010	$0.00314

Measured 20d ago against content hash ac95f9d1c688, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

datachain-knowledge scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 20d ago.

The scan reads SKILL.md. This mod also ships 18 executable files (__init__.py, collect.py, scripts/__init__.py, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

src/datachain/skill/knowledge/SKILL.md · 283 lines

How it starts

The opening of the file, as written. The whole thing — 283 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Maintain a knowledge base at dc-knowledge/. .md files are the persistent output. .json files are intermediate (generated in Step 3, consumed in Step 4, then deleted).

{core_skill_dir}/SDK.md owns how the pipeline code is written — dataset shape, row grain, provenance, naming. This file owns the knowledge base and how the work is run: what already exists, what a run will cost, and where the results land.

Critical Rules

Path is dc-knowledge/ — NOT .datachain/. The .datachain/ directory is the internal database; the knowledge base lives at dc-knowledge/.
Never pass update=True to dc.read_storage() in query or exploration code unless the user explicitly asks to refresh the listing. Build scripts that read storage are the exception — they pass update=True, delta=True.
Prefer DataChain operations over plain Python for all metadata analysis.
Bounded output — JSON and markdown files stay small regardless of data size.
Stop on auth/connection errors — bucket_scan.py runs a fast access check. If it exits with an error JSON on stderr, stop immediately and show the error to the user. Do not retry with different regions, profiles, or endpoints — ask for the missing credentials.
Follow the enrichment prompt template literally in Step 4. Downstream tooling (render_index.py) parses the exact frontmatter the prompt prescribes.

Common gotchas in UDF scripts

parallel=N vs workers=N. parallel=N is local multiprocessing (works anywhere). workers=N is Studio-only and MUST be guarded: chain = chain.settings(parallel=N); if dc.is_studio(): chain = chain.settings(workers=N).
No from __future__ import annotations in UDF modules. It stringifies type hints and DataChain's signal-schema resolution rejects the string-vs-class mismatch.
Type the UDF return precisely. Iterator[object] / Iterator[Any] / bare dict fail schema resolution. Return a specific Iterator[T], a Pydantic BaseModel, or a primitive.
Generators aren't subscriptable. Iterators returned by file APIs do not support [:N]. Use enumerate + break, or list(...) only when the result is genuinely small.
Use datachain.__version__ to get the package version (e.g. dc.__version__).

Read the full file on GitHub · 283 lines

Files

What ships with it

20 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

20d ago Changed · +64 lines ac95f9d1c688
1mo ago Changed f67f773d6b6a
1mo ago First seen · 219 lines · 104 tokens per session scan A de52962a5066

Subscribe to this mod's changes

datachain-knowledge is a skill published in the GitHub repository datachain-ai/datachain (2,826 stars, last pushed today), licensed Apache-2.0. It adds 104 tokens to every session and 3,140 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Claude Code Data Processing Data and AI MLOps codex harness-engineering unstructured-data

tensorboard

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit.

davila7/claude-code-templates · 32 tokens

nemo-curator

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or…

davila7/claude-code-templates · 83 tokens

ray-data

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

davila7/claude-code-templates · 70 tokens

rocketride-building-pipelines

Use when asked to build, run, or validate a RocketRide pipeline (e.g. "build me a chatbot that answers from my docs", "make a pipeline that summarizes uploaded PDFs", "run this pipeline"), or whenever turning a plain-language data/AI task into a working RocketRide pipeline.

rocketride-org/rocketride-server · 66 tokens

rocketride-configuring-pipelines

Use when filling in a RocketRide pipeline's node configuration and validating it — fetching each node's schema, setting required fields, running the anti-pattern checklist, and validating against the engine before a run.

rocketride-org/rocketride-server · 47 tokens

experiment-tracking-swanlab

Provides guidance for experiment tracking with SwanLab. Use when you need open-source run tracking, local or self-hosted dashboards, and lightweight media logging for ML workflows.

Orchestra-Research/AI-Research-SKILLs · 40 tokens