content-hash-cache-pattern

Content-Hash File Cache Pattern

Cache expensive file-processing results using a SHA-256 hash of the file's content as the cache key. Unlike path-based caching, this survives file moves and renames, and auto-invalidates the moment content changes — no separate invalidation logic needed.

When to Activate

• Building a file-processing pipeline (PDF parsing, OCR, text extraction, image analysis) where the same files reappear across runs
• Processing cost is high enough that reprocessing unchanged files is wasteful
• Adding a --cache/--no-cache flag to a CLI tool
• Retrofitting caching onto an existing pure function without modifying that function

Core Pattern (Go)

1. Content hash as the cache key

Hash the file in chunks — never load the whole file into memory:

package cache

import (
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"io"
	"os"
)

func ComputeFileHash(path string) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", fmt.Errorf("open %s: %w", path, err)
	}
	defer f.Close()

	h := sha256.New()
	if _, err := io.Copy(h, f); err != nil {
		return "", fmt.Errorf("hash %s: %w", path, err)
	}
	return hex.EncodeToString(h.Sum(nil)), nil
}

io.Copy streams the file into the hasher in fixed-size buffers, so this scales to large files without loading them fully into memory.

2. Cache entry type

type CacheEntry struct {
	FileHash   string   `json:"file_hash"`
	SourcePath string   `json:"source_path"`
	Document   Document `json:"document"` // the cached result
}

3. File-based storage: {hash}.json

O(1) lookup by hash, no separate index file:

package cache

import (
	"encoding/json"
	"fmt"
	"os"
	"path/filepath"
)

func WriteEntry(cacheDir string, entry CacheEntry) error {
	if err := os.MkdirAll(cacheDir, 0o755); err != nil {
		return fmt.Errorf("create cache dir: %w", err)
	}
	data, err := json.Marshal(entry)
	if err != nil {
		return fmt.Errorf("marshal cache entry: %w", err)
	}
	path := filepath.Join(cacheDir, entry.FileHash+".json")
	return os.WriteFile(path, data, 0o644)
}

// ReadEntry returns (entry, true) on a cache hit, (zero, false) on a
// miss — including a miss on corrupted or unreadable cache files, so
// corruption degrades to a re-process rather than a crash.
func ReadEntry(cacheDir, fileHash string) (CacheEntry, bool) {
	path := filepath.Join(cacheDir, fileHash+".json")
	data, err := os.ReadFile(path)
	if err != nil {
		return CacheEntry{}, false
	}
	var entry CacheEntry
	if err := json.Unmarshal(data, &entry); err != nil {
		return CacheEntry{}, false
	}
	return entry, true
}