etl-handler

Skill: etl-handler — canonical Go ETL extraction pattern

Use when implementing file-type handlers, extractors, decoders or transforms for the corpus pipeline. Mandatory before writing such code (AGENTS.md, code standards #8–#11).

Core idea: any format may nest any other format (.eml → zip → .eml → pdf). One handler per type; a central registry recurses; results are typed trees of URL-addressable nodes.

Contracts

// Blob is an addressable unit of content flowing through the pipeline.
type Blob struct {
	SHA256 string        // integrity + idempotency key
	Name   string        // sanitized filename
	MIME   string        // declared × sniffed (mismatch wins for safety)
	Size   int64
	Open   func(ctx context.Context) (io.ReadCloser, error) // lazy
}

type Attachment = Blob // child inside a Result

type Result struct {
	MIME     string      // resolved type of this node
	Meta     Meta        // typed per-handler metadata (struct, not map)
	Text     []byte      // extracted text/markdown when applicable
	Children []Attachment // nested parts — recurse via Registry
}

type Handler interface {
	Match(declaredMIME string, sniff []byte) bool
	Handle(ctx context.Context, src Blob) (*Result, error) // sync, ctx-first
}

• Node identity: URL scheme://platform/thread/msg/path-segments[#anchor] (e.g. mail://gmail/T42/M17/body/p[3]/table[0]#r2,c5); node ID = sha256(URL)[:16]; separate sha256(body) for integrity.
• Parts reference each other only via URLs; href/src become LINKS_TO edges.
• Content splits granularly BEFORE insertion: html/md/pdf yield mime-typed blocks (paragraph / table / image / link / page).

Walker safety (hard limits)

• Max depth 10; per-item size cap; total-size and file-count caps;
• zip-ratio guard: uncompressed/compressed > N → refuse;
• reject names with .., absolute paths, null bytes, control chars, length > 255; collisions get -<hash8> suffix;
• large payloads spool to var/tmp (cleaned after processing); Body []byte only under the small-size threshold (~1 MB);
• per-child errors are collected — never fatal to the whole tree.