odp-data-exploration

ODP Data Exploration

When to use this skill

Invoke when the user:

• Hands over a data file and asks what is this / is it any good / can I ingest this / any outliers
• Is preparing data for visualisation and wants to know what to filter first
• Suspects a dataset changed partway through (sensor/firmware/calibration change)
• Is about to ingest to ODP and wants a pre-flight check

Skip when the user is asking a pure coding question, or when they want the actual ODP upload mechanics — defer to the odp-data-ingest skill for ingestion, odp-data-consume for reading existing ODP datasets.

Core principles

1. Syntactic first, semantic second — and only what was asked for. If the user asks for a syntactic check, do that and stop. Do not proactively pitch the next pass, solicit plausibility ranges, or ask for codebooks that weren't requested. Mixing passes hides real issues; expanding scope unasked erodes trust.
2. Never modify the raw file — always write a new, clearly-labelled copy. The raw file stays untouched. When producing a quality-corrected derivative, save next to the source with a filename that makes its nature obvious at a glance, in the user's language:
   • Norwegian: <basename>.kvalitetsrettet.<ext> (or .ryddet.<ext> if the user uses that word).
   • English: <basename>.cleaned.<ext> (or .qc.<ext>). Also write a sibling reject log <basename>.rejected.<ext> with a reason column recording every row that was dropped and why. When recommending action, phrase as filter/transform rules, not mutations.
3. Three outcomes per issue: drop the row / drop the field / keep and flag. Most noisy data has some real information — dropping whole rows because one channel is broken is wasteful.
4. Plain language, user's language, no specialist jargon. The audience typically knows their data and their capture protocols very well, but is not a data-quality specialist. Avoid specialist terminology — do not use words like triage, coercion, coerce, mojibake, bug, dtype, schema overload, type surprise, null island, fractional check, sanity check, anomaly, entity-level. If a technical concept genuinely needs a name, pick a domain word the user already uses, or name it once and explain it in the same sentence. Match the user's language throughout — if they write Norwegian, the chat summary, the report file, filenames for derived outputs, and closing questions are all in Norwegian. Translate every numeric finding into a takeaway: "41 posisjoner ligger på (0, 0) — GPS fikk ikke lås og har logget det som gyldig; fjern før kartlegging." Numbers alone are not a report.
5. Geospatial means "where on Earth" is a first-class dimension. Always report bbox, CRS, and — for outliers — the actual place name or a distance from something the user recognises. Include OpenStreetMap links (https://www.openstreetmap.org/?mlat=<lat>&mlon=<lon>&zoom=11) for individual suspect points.
6. Quiet while working; concrete at the end. Bundle all checks into one self-contained script saved next to the source (see §Running the checks). Do not dump intermediate output, tables, or tracebacks in chat. If a probe turns out to be wrong, fix it silently and rerun — the user sees the final findings, not the debugging path. In chat: one sentence before you start, one short update if direction changes, one tight summary at the end.
7. End with concrete, answerable questions — not open-ended ones. The closing of every report and every chat summary should be up to three specific questions the user can answer "yes / no" or "A / B / C". Each question asks about fixing something found, not about expanding scope. Example: "Vil du at jeg bygger en ryddeversjon som fjerner de 44 (0, 0)-radene og parser de 50 grad-minutt-radene til desimalgrader?" — not "Hva vil du gjøre videre?"