annotate

confide:annotate — human PII gold set + inter-annotator agreement

Humans label PII spans in a transcript; you measure how much they agree (κ) and draft an adjudicated gold from their labels. Annotators are first-class here — most of this skill is plain instructions FOR a person doing the labelling, plus a coordinator path to score it.

Privacy invariants (do not violate)

• Synthetic or consented data only. Never load a real client transcript the person did not consent to share. When in doubt, anonymize first (confide:anon) and annotate the GREEN copy.
• Names stay local. The annotator's labels (which contain real surface text spans) live in their browser and the exported JSON file on their own machine. Collect label files locally.
• Nothing PII is re-shared. Only κ / F1 / disagreement clusters travel between people if needed. The transcript text and the original PII are never re-distributed by this skill.

Bundled assets

• assets/annotator.html — zero-install browser annotation tool (EN/RU, runs offline).
• references/codebook.md — the labelling rulebook (10 PII types, direct/quasi, harm).
• references/tool-guide.md — how to drive the tool + scorer step by step.
• scripts/score_iaa.py — Cohen's/Fleiss' κ, span-F1, disagreement queue, draft gold (stdlib).
• scripts/gold_to_labels.py — turn an existing gold into a "reference annotator" to test solo.

FOR THE ANNOTATOR (no coding needed)

1. Open the tool. Double-click assets/annotator.html (or open it in Chrome/Firefox/ Safari). It runs entirely in your browser — nothing is uploaded; labels stay on your machine until you Export.
2. Read the rules. Open references/codebook.md first. It defines the 10 types (PERSON, LOCATION, ORG, PHONE, EMAIL, ID, DATE, MEDICATION, AGE, PROFESSION), what counts as a span (the minimal identifying text), and direct vs. quasi-identifier.
3. Set your annotator id and load the transcript in the tool (e.g. A, B, or your name). Use only synthetic or consented text.
4. Label every PII span. Select the minimal text that identifies a real person (the client or third parties they mention) and assign its type. Record direct/quasi, entity id, role, and harm as the codebook describes. Do not rewrite or redact — only label.
5. When unsure, log it — don't guess silently. Add a note starting with QUESTION: on the span (e.g. QUESTION: gym or city?). These flow straight into the adjudication queue.
6. Export. Click Export → you get labels.<doc>.<annotator>.json (schema: {doc_id, annotator, text, spans:[{start,end,text,type,...}]}). Keep it local and hand only this file to the coordinator. Two+ people should label the same doc independently (blind) for a meaningful κ.